Trouble using conversion through JSON about openrtb-doubleclick HOT 18 CLOSED

I see that you're trying to use GSON (or any automatic JSON mapping API like Jackson's ObjectMapper) to serialize the content of this library's types like OpenRtb.BidRequest. This will not work well because protoc-generated classes are not proper POJOs; in addition to fields terminated by '_' they have non-JavaBean-standard method names, they have other fields for things like absent/present bits, the UnknownFieldSet, extensions... so you are likely to have other problems in the generated JSON unless you write more workarounds. I just don't understand why are you trying to do this serialization with GSON, when you could use the included OpenRtbJson* API?

The booleans converted to YES / NO are easy to explain: first, the OpenRTB specification doesn't really specify any API, only a JSON format which we do implement strictly (the generated JSON validates correctly with Nexage's OpenRTB Validator, but let me know if you see any exception). The concrete API that we define with protobuf fills in some gaps or fixes some issues, like the use of 0/1 values for many boolean flags without any good reason (other than the fact that 0/1 use less space in JSON text...). But the OpenRtbJson* API will correctly serialize these flags as 0/1.

from openrtb-doubleclick.

To add a bit to this issue, I've tired my own FieldNamingStrategy on my GSON object:

new GsonBuilder().setFieldNamingStrategy(new FieldNamingStrategy() {
    @Override
    public String translateName(Field f) {
        String name = f.getName();
        if (name.endsWith("_")) {
            return name.substring(0, name.length() - 1);
        }
        return name;
    }
}).create();

I then get alot of format exceptions. For instance in the field pos there's a String ("POSITION_UNKNOWN") although what OpenRTB expects is an Integer. I see also that alot of booleans are defined as strings using "NO" instead of false. Is this behavior expected on your end ? Is this going to evolve ?

Would it be possible to adapt those models so that they reflect the OpenRTB naming standards as well as typing ? Or maybe give a standard way to obtain a result that respects the OpenRTB specs ?

Thanks

from openrtb-doubleclick.

Indeed you're right. I misread the text concerning the OpenRtbJson* API and ended up discarding it when infact it was what I needed. I'll close this :)

from openrtb-doubleclick.

It seems I'm still having some problems even using the openrtb-json lib but I think these problems are due to the differences between DC and OpenRTB.

Once my BidResponse is converted to OpenRtb.BidResponse, i get the following validation errors:

Problems:
        Response contains no processing time information.
        Ad 0: ad does not contain any click-through urls.
        HTML snippet for ad 0: click url macros missing (CLICK_URL_UNESC or CLICK_URL_ESC)

I understand DC doesn't support the win url so i'll have to add it to the ad markup. There is also no field that may correspond to the click-through url in OpenRtb or the processing time so i guess that's why those are missing.

After reading the wiki, I tried to come up with a solution using the ext field as it's suggested in the announcement of the OpenRTB & OponRTB-DoubleClick libraries but so far I couldn't make a value from my own BidResponse's ext up to the native bid response click_through_url.

I got this working:

    @Test
    public void test_1() throws IOException {
        OpenRtb.BidResponse.Builder b2 = OpenRtb.BidResponse.newBuilder();
        b2.addSeatbidBuilder().addBidBuilder()
            .setId("1")
            .setImpid("1")
            .setCrid("creative-123")
            .setAdm("...lots of HTML goes here...")
            .setPrice(1.0)
            .setExtension(DcExt.ad, Doubleclick.BidResponse.Ad.newBuilder().addClickThroughUrl("http://lalala.com").build());
        OpenRtb.BidResponse brs = b2.build();

        Assert.assertEquals("http://lalala.com", GoogleAdxMapper.get().toNativeBidResponse(getBidRequest(), brs).getAd(0).getClickThroughUrl(0));
    }

but that only uses your OpenRTB classes so i tried starting from mine and converting it but whatever I put in the ext field, I get a java.io.IOException: Unhandled extension.

Any idea what I'm missing ? Is there a way to set the missing required fields in my object so it's recognized in the end by the openrtb-json deserializer ?

Thanks

from openrtb-doubleclick.

It seems you are using the google-rtb-requester which generates these validations. Well, first the processing time, this is an optional fields in the DoubleClick response, so you can just ignore this warning... if it's not just a warning and google-rtb-requester fails because of that problem, it's a bug that should be reported there. Or you can just set any value for this field. The DoubleClickOpenRtbMapper returns a builder so you can still change fields before calling build(); the OpenRtbJsonReader returns the "built" object in most readBidResponse() overloads, but these are just conveniences, you may call the main method readBidRequest(JsonParser par) that will return a builder (or just call the others but use toBuilder(), it's not very expensive because all properties are immutable so it's a shallow-copy).

For the clickthrough URL you are on the right path. This is the biggest problem in OpenRTB / Doubleclick integration because this field is required (the protobuf specifies it as optional, but the exchange will reject the bid if it's not set). You code snippet doing the setExtension(...) is right, I was able to run it successfully here; I think your's only missing correct setup of the mapper object. The mapper has a modular architecture, so the core mapper object doesn't handle any extensions, not even DcExt; you need to configure it to do that like this (from DoubleClickOpenRtbMapperTest):

private DoubleClickOpenRtbMapper mapper = new DoubleClickOpenRtbMapper(
    new MetricRegistry(),
    TestUtil.getMetadata(),
    new DoubleClickCrypto.Hyperlocal(TestUtil.KEYS),
    ImmutableList.of(DoubleClickLinkMapper.INSTANCE));

You're certainly missing the last parameter, it's a list of extension handlers where you need to include DoubleClickLinkMapper.INSTANCE to handle DcExt. Hope this fixes the final issues :)

from openrtb-doubleclick.

Indeed, this last parameter was empty. I just defined it as you suggested.
However i'm still failing.
I noticed that DoubleClickLinkMapper never uses DcExt.ad though it uses all the other DcExt.*. Maybe that's cause DcExt.ad is from the BidResponse rather than the BidRequest ?

Also, I wouldn't know how I should define the ext field of my BidResponse object. Should i just do something like:

brs.ext = Doubleclick.BidResponse.Ad.newBuilder().addClickThroughUrl("http://lalala.com").build();

Or should i wrap this up in a Map ? If so what would then be the key ?
I've failed in trying most of those combinations yet.

FYI, I would then json-encode my BidResponse (using gson this time as openrtb-json only provides encoding/decoding at the top level of objects) then json-decode it using openrtb-json to get an OpenRtb.BidResponse. Does this seem valid to you ?
Currently this leaves me with a JSON that looks bad to me as my ext contains a protoc-generated class and as you said earlier: not a proper POJO.

Thanks a lot for your help regarding these matters.

from openrtb-doubleclick.

Oops I just assumed that the DoubleClickLinkMapper was the missing piece, but you are right, it's not necessary for this particular problem. This mapper helper is only necessary to stick the DoubleClick nodes as extensions inside the OpenRTB model, which you may want for other reasons but not relevant for the rest of OpenRTB mapping. The core mapper will take care of the clickthrough URL even without that parameter. Here's a complete unit test based on your code that works for me:

@Test
public void test_1() {
  DoubleClickOpenRtbMapper mapper = new DoubleClickOpenRtbMapper(
      new MetricRegistry(), TestUtil.getMetadata(),
      new DoubleClickCrypto.Hyperlocal(TestUtil.KEYS),
      ImmutableList.<ExtMapper>of());
    OpenRtb.BidResponse.Builder b2 = OpenRtb.BidResponse.newBuilder();
    b2.addSeatbidBuilder().addBidBuilder()
        .setId("1")
        .setImpid("1")
        .setCrid("creative-123")
        .setAdm("...lots of HTML goes here...")
        .setPrice(1.0)
        .setExtension(DcExt.ad, Doubleclick.BidResponse.Ad.newBuilder()
            .addClickThroughUrl("http://lalala.com").build());
    OpenRtb.BidResponse brs = b2.build();
    BidRequest request = TestUtil.newBidRequest(TestData.newRequest());
    assertEquals(
        "http://lalala.com",
        mapper.toNativeBidResponse(request, brs).getAd(0).getClickThroughUrl(0));
}

The TestUtil and TestData above are from doubleclick-openrtb's unit tests, so you may want to copy that to start with a test that (hopefully) works, then find what's different from your code.

from openrtb-doubleclick.

Sorry I may have not be precise enough. The test case you're presenting here works fine as was the one i presented before.

My problem is that I'm adding doubleclick-openrtb to my existing bidder that already works in a full openrtb way.
Currently, i'm receiving JSON bid requests (from other SSPs than google) which I json-decode to my own BidRequest object. I then construct my own BidResponse object that I json-encode and send back as a response.
My goal here is to use doubleclic-openrtb to easely integrate dc as a new ssp in our current code.

I've currently successfully translated the incoming dc native BidRequest to my own BidRequest object thanks to your help by doing the following:

1. Create a dc native BidRequest
2. Convert it to a dc openrtb BidRequest
3. json-encode it using openrtb-json
4. json-decode it to *my own * BidRequest object using Gson.

Now our current code processes this BidRequest and outputs a BidResponse of my own that i need to transform to a dc native BidResponse
What I fail to do is the following

1. Set the click_through_url in the ext field of my own BidRespone.SeatBid.Bid
2. json-encode it using Gson
3. json-decode it to a dc openrtb BidResponse using openrtb-json
4. Convert it to a dc native BidResponse
5. Validate that the field click_through_url is indeed set.

The transformation works but I fail to pass the click_through_url accross the objects due to the json-encode / json-decode part.

Thanks a lot for your support.

from openrtb-doubleclick.

Oh I understand better now. What you are trying to do is difficult because the "other" OpenRTB library does not support the same extensions. Can you post here the JSON form of the BidResponse emitted by this other library, including its clickthrough extension?

from openrtb-doubleclick.

I'll post that tomorrow if you really need it as I do not have access to this right now but it's really standard. You can imagine it's a pure OpenRTB BidResponse (as the ones shown in the examples of the IAB OpenRTB API Specs) and that I can put whatever I want in the ext field. I'd just need to know what I have to put there so that it's understood by the openrtb-json lib.

from openrtb-doubleclick.

Ok, so you have a bid with an extension like ext { "click_through_url": "http://lalala.com" }. One way to solve this is once again with a plugin, but now in the openrtb-json's decoder. This means implementing the OpenRtbJsonExtReader interface, and also creating a protobuf that defines the extension. The protobuf is trivial, here's what you need, let's call this MyExt:

message Bid {
  repeated string clickThroughUrl = 1;
}
extend com.google.openrtb.BidResponse.SeatBid.Bid {
  optional Bid bid = 110;
}

Then the JSON serializer plugin, look at Test1Reader inside OpenRtbJsonTest but extending OpenRtbJsonExtReaderBase<OpenRtb.BidResponse.SeatBid.Bid.Builder, Doubleclick.BidResponse.Ad.Builder>, and writing a case "click_through_url". Then you configure this plugin with OpenRtbJsonFactory.register(), and the resulting reader object will be able to parse this extension into our OpenRTB model. But that's not done yet, you also need to implement an ExtMapper to configure the DoubleClickOpenRtbMapper to map this enriched model to the DoubleClick native model.

It sounds like a lot of work, but these extensibility APIs are designed for more ambitious work like implementing a full "exchange connector" that handles an arbitrary exchange's messages in any format. For a more limited problem like yours it's a bit overkill, at least the learning curve (the end result won't have much code, since you only have a single extension field). Documentation for this could be improved... right now you could check the docs for Open Bidder (https://developers.google.com/ad-exchange/rtb/open-bidder/guides/x-exchange-guide) but I hope to eventually have more documentation and code samples of this kind, for this library, without dependencies on the Open Bidder project.

But here's a trick to make this much easier: don't write an extension protobuf or an ExtMapper implementation. Just adopt the DcExt, creating a serializer plugin like this:

static class MyReader extends OpenRtbJsonExtReaderBase<
    OpenRtb.BidResponse.SeatBid.Bid.Builder, Doubleclick.BidResponse.Ad.Builder> {
  public MyReader() {
    super(DcExt.ad, Doubleclick.BidResponse.Ad.newBuilder());
  }
  @Override public boolean read(
    EB msg, Doubleclick.BidResponse.Ad.Builder ext, JsonParser par) throws IOException {
    switch (getCurrentName(par)) {
      case "click_through_url":
        for (OpenRtbJsonUtils.startArray(par); OpenRtbJsonUtils.endArray(par); par.nextToken()) {
          ext.addClickThroughUrl(par.getText());
        }
        return true;
      default:
        return false;
    }
  }
}

Then register this reader in the factory with OpenRtbJsonFactory.register(new , "BidResponse.seatbid.bid"), and create the mapper with the DoubleClickLinkMapper. Now everything should work, hopefully. This special DcExt extension is not designed to map DoubleClick's OpenRTB extensions (because such thing doesn't exist – Ad Exchange has no support to sending bid requests or receiving bid responses in OpenRTB format), so it's only good for in-memory only usage, not for any on-the-wire messaging (that's the reason why the library doesn't provide JSON serialization support for it). But your problem looks like just the kind of scenario that can use this. The full Doubleclick.BidResponse.Ad object is more than you need since you will only use the click_through_url, but this causes no problem since this extension object will be only a temporary holder.

from openrtb-doubleclick.

Ok so I did as you suggested.

Here's a recap in case it serves someone else:

new DoubleClickOpenRtbMapper(
    new MetricRegistry(),
    new DoubleClickMetadata(new ResourceTransport()),
    null, //new DoubleClickCrypto.Hyperlocal(new DoubleClickCrypto.Keys()),
    ImmutableList.<ExtMapper> of(DoubleClickLinkMapper.INSTANCE));

The static class:

static class MyReader extends OpenRtbJsonExtReaderBase<OpenRtb.BidResponse.SeatBid.Bid.Builder, Doubleclick.BidResponse.Ad.Builder> {
    public MyReader() {
        super(DcExt.ad, Doubleclick.BidResponse.Ad.newBuilder());
    }

    @Override
    protected boolean read(Builder msg, com.google.doubleclick.Doubleclick.BidResponse.Ad.Builder ext, JsonParser par) throws IOException {
        switch (OpenRtbJsonUtils.getCurrentName(par)) {
            case "click_through_url":
                for (OpenRtbJsonUtils.startArray(par); OpenRtbJsonUtils.endArray(par); par.nextToken()) {
                    ext.addClickThroughUrl(par.getText());
                }
                return true;
            default:
                return false;
        }
    }
}

And my test:

    @Test
    public void test_2() throws IOException, EncrypterException {

        OpenRtbJsonFactory  jsonFactory = OpenRtbJsonFactory.create().register(new MyReader(), "BidResponse.seatbid.bid");

        BidResponse mybrs = new BidResponse();
        SeatBid myseatbid = new SeatBid();
        Bid mybid = new Bid();
        mybid.id = "1";
        mybid.impid = "1";
        mybid.crid = "creative-123";
        mybid.adm = "...lots of HTML goes here...";
        mybid.price = 1.0f;

        HashMap<Object, Object> ext = new HashMap<Object, Object>();
        ext.put("click_through_url", new String[] { "http://lalala.com" });
        mybid.ext = ext;

        myseatbid.addBid(mybid);
        mybrs.addSeatBid(myseatbid);

        String json = JsonTranslator.toJson(mybrs); // uses Gson
        System.out.println(json); // {"seatbid":[{"bid":[{"id":"1","impid":"1","price":1.0,"adm":"...lots of HTML goes here...","crid":"creative-123","ext":{"click_through_url":"http://lalala.com"}}]}]}
        OpenRtb.BidResponse brs = jsonFactory.newReader().readBidResponse(json);

        Assert.assertEquals("http://lalala.com", GoogleAdxMapper.get().toNativeBidResponse(getBidRequest(), brs).getAd(0).getClickThroughUrl(0));
    }

It works perfectly :) I first had the json-decode part hanging forever because i set a String rather than an array of String in the click_through_url field.

Wouldn't you benefit from making this part of your lib though ? I think it may just be what's needed so that one can fully integrate an existing system with doubleclick. One could even imagine adding the doubleclick integration in a middleware

          +-----------+          
          |Doubleclick|          
          +-+-------^-+          
            |       |            
     dc-brq |       | dc-brs     
            |       |            
          +-v-------+-+          
          |Middleware |          
          +-+-------^-+          
            |       |            
openrtb-brq |       | openrtb-brs
            |       |            
          +-v-------+-+          
          |Bidder     |          
          +-----------+          

Unless you wish to consider this a a new feature request you may close this.
Thanks a lot for your help. You've been really useful and this lib is really great. Saved us so much time :)

from openrtb-doubleclick.

One last thing: I tried to do the same with the ext field of the root BidResponse object and it doesn't seem to work as there is no DcExt.bidResponse defined.

java.lang.IllegalStateException: getDescriptor() called before internalInit()
    at com.google.protobuf.GeneratedMessage$GeneratedExtension.getDescriptor(GeneratedMessage.java:1301)
    at com.google.protobuf.GeneratedMessage$ExtendableBuilder.verifyExtensionContainingType(GeneratedMessage.java:894)
    at com.google.protobuf.GeneratedMessage$ExtendableBuilder.hasExtension(GeneratedMessage.java:909)
    at com.google.openrtb.json.OpenRtbJsonExtReaderBase.read(OpenRtbJsonExtReaderBase.java:65)
    at com.google.openrtb.json.OpenRtbJsonReader.readExtensions(OpenRtbJsonReader.java:1036)
    at com.google.openrtb.json.OpenRtbJsonReader.readBidResponse(OpenRtbJsonReader.java:939)
    at com.google.openrtb.json.OpenRtbJsonReader.readBidResponse(OpenRtbJsonReader.java:899)
    at com.google.openrtb.json.OpenRtbJsonReader.readBidResponse(OpenRtbJsonReader.java:892)

Do you think this could be added so that we can map the OpenRTB.BidRequest.ext to the Doubleclick.BidResponse.processing_time_ms field using the same technique as you shown me above ? I know this field is optional but i'd like to be thorough.

from openrtb-doubleclick.

There's no DcExt.bidResponse just because we didn't anticipate the need, but your scenario is certainly reasonable. Adding that in the next version.

from openrtb-doubleclick.

Done, you can get this now from the fork https://github.com/opinali/openrtb-doubleclick if you can't wait for the next release here.

I will definitely consider what I can improve to make scenarios like yours easier (at the very least, there's some work to do in wikis/docs).

from openrtb-doubleclick.

You're just awesome ;)
Indeed, improving the wikis/docs will help a lot. I think almost everything can be done but one need to know how :)
Thanks !

from openrtb-doubleclick.

Hi,

It seems the approach to solve the problem is not valid anymore for the latest version (0.8.3). Any hint to achieve it with the latest version?

Thanks,

from openrtb-doubleclick.

Hi,

Here is the working implementation for the latest version (0.8.4).

Mapper:

import com.fasterxml.jackson.core.JsonParser;
import com.google.doubleclick.DcExt;
import com.google.openrtb.json.OpenRtbJsonExtReader;
import com.google.openrtb.json.OpenRtbJsonUtils;
import com.google.protobuf.GeneratedMessage;
import com.google.protos.adx.NetworkBid;

import java.io.IOException;

public class ClickThroughUrlMapper<EB extends GeneratedMessage.ExtendableBuilder<?, EB>> extends OpenRtbJsonExtReader<EB, NetworkBid.BidResponse.Ad.Builder> {

    public ClickThroughUrlMapper() {
        super(DcExt.ad);
    }

    @Override
    protected void read(EB msg, NetworkBid.BidResponse.Ad.Builder ad, JsonParser par) throws IOException {
        switch (OpenRtbJsonUtils.getCurrentName(par)) {
            case "click_through_url":
                for (OpenRtbJsonUtils.startArray(par); OpenRtbJsonUtils.endArray(par); par.nextToken()) {
                    ad.addClickThroughUrl(par.getText());
                }
        }
    }

}

Registration:

OpenRtbJsonFactory  jsonFactory = OpenRtbJsonFactory.create().register(new ClickThroughUrlMapper(), OpenRtb.BidResponse.SeatBid.Bid.Builder.class);

The rest is the same @Crystark exposed.

Best,

from openrtb-doubleclick.