google / openrtb-doubleclick Goto Github PK

Hi,

It would be nice to have in the wiki the best practices concerning thread safeness of this library. For instance, DoubleClickMetadata is quite heavy to load seeing how many files it reads so it would be really interesting to have it as a class constant. After taking a look to the class, it looks like it's thread safe but i haven't tested it much yet.

Would the DoubleClickOpenRtbMapper be thread safe too ? What about the MetricRegistry and the DoubleClickCrypto.Hyperlocal classes ?

I think it would be great for any user to know what they can create only once and reuse and what they can't as RTB is alot about performance.

Thanks

I checked on maven repository there does not seem to be any release version of doubleclick-core 1.5.16, how can I use it?

Hi
we are in need of extension key for Deal Object in bidrequest. But could not find Deal.Ext in double click? will it be added in upcoming versions of double click? If we want to add what kind of implementation can be proceed.

Hi all,

We just noticed that the adx-openRTB mapping contains some inconsistencies with the official documentation from Google. For example, in the mapping there is an Adx code 13003 which doesn't exist in the official documentation.
Also, there are some values which exist in the official documentation but not in the mapping, for example Adx code 11950.

Can you shed some light over it?

Many thanks,
Alex & Massimo

Good day!

Is there some sort of a timeline or a due date of when the latest doubleblick-openrtb will be available in maven?

We have identified 2 possible instances of outdated documentation:

• AdxExtUtils was deleted but the change has not been reflected in the current version of the Home wiki page

• OpenRtbJsonFactory was deleted but the change has not been reflected in the current version of the Home wiki page

About

This is part of a research project that aims to automatically detect outdated documentation in GitHub repositories. We are evaluating the validity of our approach by identifying instances of outdated documentation in real-world projects.

We hope that this research will be a step towards keeping documentation up-to-date. If this has been helpful, consider updating the documentation to keep it in sync with the source code. If this has not been helpful, consider updating this issue with an explanation, so that we can improve our approach. Thanks!

I am looking for ext Object for Deal

Can anyone tell me how to create DealExt?
I am using java openrtb bid request Generation version 2.5

OpenRTB API Specification (2.x versions) uses the Geo object to transfer information on a user/device location. Usually the following Geo object fields are used to provide geo targeting:

• country—Country using ISO-3166-1 Alpha-3.
• region—Region using ISO 3166-2
• city—City using United Nations Code for Trade and Transport Locations (http://www.unece.org/cefact/locode/service/location.htm)

DoubleClick uses the geo_criteria_id field of the message BidRequest. The locations list is described in the geo-table.csv file. It includes country and region codes that meet ISO codes. However, it doesn't include city codes. Also city names don't match to UN/LOCODE city names in some cases. More over in certain cases city parent region don't match to the UN/LOCODE too. All the above mentioned makes it impossible to ensure matching between the Geo object and the geo_criteria_id field.

Some examples (UN/LOCODE vs geo-table.csv):

• Moscow (the capital of Russia) has "RU MOW" code and "Moskva" name vs "Moscow" name in geo-table.csv;
• London's parent region code is "GB-LON" vs "GB-ENG" in geo-table.csv.

Hi,

Seeing that billing_id is a String in the OpenRTB json request. However the ImpExtReader is expecting that as a Long.
https://developers.google.com/ad-exchange/rtb/openrtb-guide

Getting the below exception:

com.fasterxml.jackson.core.JsonParseException: Current token (VALUE_STRING) not numeric, can not use numeric value accessors at [Source: org.glassfish.grizzly.utils.BufferInputStream@4ecbbe0b; line: 1, column: 163] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419) ~[jackson-core-2.4.3.jar:2.4.3] at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508) ~[jackson-core-2.4.3.jar:2.4.3] at com.fasterxml.jackson.core.base.ParserBase._parseNumericValue(ParserBase.java:748) ~[jackson-core-2.4.3.jar:2.4.3] at com.fasterxml.jackson.core.base.ParserBase.getLongValue(ParserBase.java:619) ~[jackson-core-2.4.3.jar:2.4.3] at com.google.doubleclick.openrtb.json.ImpExtReader.read(ImpExtReader.java:46) ~[?:?] at com.google.doubleclick.openrtb.json.ImpExtReader.read(ImpExtReader.java:35) ~[?:?] at com.google.openrtb.json.OpenRtbJsonExtComplexReader.readSingle(OpenRtbJsonExtComplexReader.java:93) ~[?:?]

{ "id": "V+u4DAAI8CEKG+PVxAwrsA", "imp": [ { "id": "1", "banner": { "w": 320, "h": 480, "pos": 1, "api": [ 3, 5 ], "wmax": 360, "hmax": 640, "wmin": 320, "hmin": 280 }, "instl": 1, "tagid": "1", "bidfloor": 0.01, "bidfloorcur": "USD", "ext": { "billing_id": [ "12345678901" ], "publisher_settings_list_id": [ "12345678901234567890" ], "allowed_vendor_type": [ 113 ] } } ], "app": { "name": "Super Speed VPN", "bundle": "com.example.free.vpn.proxy", "publisher": { "id": "1" }, "content": { "url": "https://play.google.com/store/apps/details?id=com.example.free.vpn.proxy", "contentrating": "DV-G", "userrating": "4.2", "language": "en" } }, "device": { "ua": "Mozilla/5.0...(Mobile; afma-sdk-a-v9683236.8115000.2)", "ip": "123.456.789.0", "geo": { "country": "FRA", "utcoffset": 180 }, "make": "alcatel", "model": "5022d", "os": "android", "osv": "5.1", "devicetype": 4, "ifa": "ec162290-a098-43ac-9fac-411861ba2b1a", "w": 360, "h": 640, "pxratio": 2 }, "user": { "id": "CAESEPfSJVjdc8Mx2U_Dd3fQqvM" }, "cur": [ "USD" ] }

Is there a timeline for when this repo will incorporate openrtb-adx.proto v28 and thusly the GDPR consent fields?

https://developers.google.com/ad-exchange/rtb/downloads/openrtb-adx-proto

I write python api to receive the bid request from google double click , of course ,it is json format .But now I have to change the format from json to protobuf when send response or receive bid requests.
What do i do? I do not know how to write the code after reading the doc of https://developers.google.com/authorized-buyers/rtb/openrtb-guide.
Obviously， google developers' doc is not work. it does not list the details about this situation(change the format from json to protobuf when send response or receive bid requests. ).
So ,i need help about above.

Hi,

I have some concerns with this two fields. The mapping from "BidRequest.Mobile.carrier_id" to "OpenRtb.Device.carrier" leaves the carrier id provided by google, but that is not very useful if you don't convert it to the carrier name or standard code. I was wondering if it makes sense that this conversion would be covered by the DoubleClickMetadata class, like it is done with other CVS dictionaries. Specially because there is a link in the BidRequest message documentation pointing to a google hosted dictionary that covers that conversions (https://developers.google.com/adwords/api/docs/appendix/mobilecarriers).

The case of app categories is similar. In the DoubleClick <-> OpenRTB Mapping spreadsheet (https://docs.google.com/spreadsheets/d/1ormTUJziNQGnGEJcZJ4--NTa1IKi2dfgfzxBhc4oe9w) it is said that "DC.BidRequest.Mobile.app_category_ids" maps to "ORTB.App.cat", but that's not implemented in the library. Moreover the "BidRequest.Mobile.app_category_ids" documentation points to a dictionary with the google app categories IDs and names, but it does not provide a mapping to standard IAB names, which are the contents that should be stored in the ORTB.App.cat field. Maybe it would be useful to have a dictionary that maps App categories to IAB Content categories in the DoubleClickMetadata class too.

Thanks.

Hi,

Similarly to #110 both (device.geo.lat and device.geo.lon) should be mapped to double instead of string
More:
https://developers.google.com/ad-exchange/rtb/openrtb-guide or any openRTB spec

Hello,

Thanks for the amazing library and documentation. I'd like to report a small issue.

According to your wiki (https://github.com/google/openrtb-doubleclick/wiki)

These protocols define dozens of message types and hundreds of properties, so a practical way to document every single mapping is with a big spreadsheet.

This "big spreadsheet" is linked via http://goo.gl/6rOifS which seems to be broken since it says "Sorry, the file you have requested does not exist."

Could you please fix it? This spreadsheet is very useful and it's pity not to have it online:)

Appreciate your help.

Good day!

Is there a timeline for the 2.0.2 version?

I especialy interested in the SKAdNetwork fields available without forking.

After integration I found some strange values of decrypted idfa/aaid values (encrypted_advertising_id). It's supposed that idfa and aaid are 32 chars hexadecimal with 4 hyphens (8-4-4-4-12).

What I am seeing after decrypt is a 32 chars hexadecimal without any hyphen, is it an idfa/aaid without hyphens or am I missing something else?

This is my code used to generate bidRequest
.setExtension(AdxExt.imp, AdxExt.ImpExt.newBuilder().setDfpAdUnitCode(adBlockInfo.getAdunitId().toString()).build()))

when I executed I get ImpExt object as
[com.google.doubleclick.imp] {
dfp_ad_unit_code: "1ad35672-f820-47c8-91dc-bf8b08410cbd"
}

How can I change "[com.google.doubleclick.imp]" to "ext"

can anyone help to solve this issue?

details address:https://groups.google.com/forum/#!search/protobuf$20openrtb$20$20|sort:date/protobuf/JCiZi3JVumc/gs61xoQ1BgAJ

in response ,I format data to protobuf in openrtb2.0 (proto2) and python 3.7. but the support team said the seatbid and bid attr 's value is [](array or list ) not '{}'.
how to change my code? the seatbid and bid attr is array(repeated) in proto file.

Hi,

We've noticed that for postal codes entities the canonical name field in the geo-table doesn't include the city. For example:
In the bid request we're getting:
postal_code: "33611"
geo_criteria_id: 9012102
This maps to the following entry in the geo-able:
9012102,"33611","33611,Florida,United States","1015214,9040475,9057273,200539,21142,2840","","US","Postal Code"
In this case the GeoObject will not set the city attribute, which is Tampa (ID 1015214).
The implementation of the library seems to be using the canonical name and not looking at the parents column in order to fill the city, region and metro.

Is this planned to be supported? Is there a workaround for city targeting?

In the AdX pretargeting UI and in the Adwords UI I see that it does the matching between the zipcode and the city.

Thanks,
Avi.

We have a OpenRTB response JSON and converted it to
OpenRtb.BidResponse.Builder

Now how will we send back response back to AdX for OpenRTB protobuf ?

OpenRtb.BidResponse.Builder has been constructed from openRTB JSON.
I believe we need to send NetworkBid.BidResponse back to AdX. Am i correct ? if yes, what would the way to create NetworkBid.BidResponse from OpenRtb.BidResponse.Builder ?

And how to use DoubleClickValidator to validate the response before we send it back to AdX ?

it seems perfect to provide a java example of decode the winning price. however it seems difficult to decode the price use golang. so wish to add golang version. That will be more perfect

Hey @opinali ,

Notice two things related to video ads:

1. DoubleClick specify values of video min and max duration in milliseconds and OpenRtb is in seconds.

DoubleClickOpenRtbMapper.java

  protected Video.Builder buildVideo(... {
    ...
    if (dcVideo.hasMinAdDuration()) {
      video.setMinduration(dcVideo.getMinAdDuration());
    }
    if (dcVideo.hasMaxAdDuration()) {
      video.setMaxduration(dcVideo.getMaxAdDuration());
    }
    ...
}

2. Despite not having any significant collateral effect due to both specs using the same values, when mapping the video start delay, you are using the toDoubleClick method instead of toOpenRtb

DoubleClickOpenRtbMapper.java

  protected Video.Builder buildVideo(... {
    ...
    if (dcVideo.hasVideoadStartDelay()) {
      video.setStartdelay(VideoStartDelayMapper.toDoubleClick(dcVideo.getVideoadStartDelay()));
    }
    ...
}

Hi,

First of all, I want to thank you for this amazing library and documentation.

I ran into the very same problem described here and found that issue to know how to solve it.

Now, my understanding is that DoubleClick's "click_through_url" field has the same purpose that the OpenRTB's "adomain" field. So, if I am not wrong, it would be nice to modify DoubleClick's mapper to map these two fields. What to you think?

Best,

Hi,

It's been a while since I've updated the version of openrtb-doubleclick we use but as we're facing some issues with the App/Site distinction I'm trying to do so now.
It seems like the code I had isn't directly compatible with 1.0.0 so had a few questions:

1- DoubleClickOpenRtbMapper used to be instantiated with a DoubleClickCrypto.Hyperlocal as parameter. I see it's not anymore. Is it now not useful anymore or am I missing something ?

2- I used to register on the OpenRtbJsonFactory a OpenRtbJsonExtReader named BidExtReader that would add the click_through_url field using the NetworkBid.BidResponse.Ad.Builder.addClickThroughUrl method (see #13). It looks like this is no longer required and that the click_through_url is now mapped from the adomain field. Can you confirm that ?

Thanks
Regards

Hi,

Is it possible to have the bid request automatically use the DoubleClickMetadata object ? I'm interested in having the carrier names instead of the google ids in my parsed bid request.
Thanks

The AdxExt has support for setting bid.ext.attribute (via e.g. the AdxExt.BidExt.Builder.addAttribute() method), but when converting it to JSON the attributes set is not included.
Looking at the BidExtWriter and BidExtReader one can see that it contains code for reading/writing the other bid.ext fields (e.g. 'impression_tracking_url'), but NOT the ''attribute' field.

I am using the openrtb-doubleclick library to convert AdX bid requests to OpenRtb requests, and am finding that it is failing to properly convert IPv4 ByteStrings. It seems that MapperUtil.toIpv4String is only iterating and converting the first three bytes in the IPv4 address, and padding the last one with a zero. The prevents DoubleClickOpenRtbMapper.toOpenRtbBidRequest from properly converting bid requests that contain IPv4 addresses.

Repro steps:
MapperUtil.toIpv4String(ByteString.copyFrom(new byte[]{(byte) 0xC6, (byte) 0x8C, (byte) 0xCC, (byte) 0x74}))

Expected: "198.140.204.116"
Actual: "198.140.204.0"

Hi Osvaldo,

First, I would like to thank you for that lib. It is being very helpful to work with it to do the mapping between DC and OpenRTB specs.

I am just confused by the way the lib creates an App or a Site. From what I know, OpenRTB states that a request coming from a non-browser application contains an App object and a request from a website contains a Site object.

While reading from DoubleClick spec, I understand that a request comes from an application when you have a Mobile object and Mobile.is_app = true, right?

But looking into the code I can read:

    if (dcRequest.hasMobile()) {
      App.Builder app = buildApp(dcRequest);
      if (app != null) {
        request.setApp(app);
      }
    } else {
      Site.Builder site = buildSite(dcRequest);
      if (site != null) {
        request.setSite(site);
      }
    }

From the code above, the lib is creating an App object when the Mobile object is present in the request, despite if is a request from the mobile web or an in-app request.

So, shouldn't be added this other condition like the code below?

    if (dcRequest.hasMobile() && dcRequest.getMobile().getIsApp()) {
      App.Builder app = buildApp(dcRequest);
      ...
    } else {
      Site.Builder site = buildSite(dcRequest);
      ...
    }

Looks like the mapping logic is there - it's just not being used to fill out the App.cat field

Hey @opinali,

As I said before, thanks for developing and maintaining this awesome libs.

Well, while working with DoubleClickOpenRtbMapper I faced the following situation after getting an instance of this class.

• This class uses the class AdCategoryMapper which provides a series of methods to convert categories between both specs (DoubleClick and OpenRtb)
• AdCategoryMapper has a static block that will be executed in the moment it is loaded by JVM (when it is first referenced in code)
• This static block is responsible of reading a text file where the mapping between categories in both specs is defined

So, the issue happens due to the number of concurrent calls to DoubleClickOpenRtbMapper (after all, it is RTB, right?). Then, as receiving multiple requests, the JVM fails to execute the static block once, and ends executing more than one time concurrently. As there's only one file, this second (or third, forth...) fails to access the mapping text file (as it is open by another thread) and a ExceptionInInitializerError is thrown. Thus, several subsequent calls to AdCategoryMapper methods end with a NoClassDefFoundError exception.

I solved this by forcing the load of the AdCategoryMapper during the init of my system:

try {
   Class.forName(AdCategoryMapper.class.getName(), true, AdCategoryMapper.class.getClassLoader());
} catch (ClassNotFoundException e) {
   throw new AssertionError(e);
}

This way, when when DoubleClickOpenRtbMapper started to receive requests to convert and used the AdCategoryMapper static methods, it already had the static block executed and had the mapped categories in its static attributes.

I thought about opening a PR with some method like initMappers() in the constructor of DoubleClickOpenRtbMapper to force the loading of mappers like AdCategoryMapper.

What do you think?

Issue:
mvn clean install -DskipTests

[INFO] Google DoubleClick Libraries ....................... SUCCESS [  0.577 s]
[INFO] Google DoubleClick Core ............................ SUCCESS [ 13.613 s]
[INFO] Google DoubleClick OpenRTB ......................... FAILURE [  0.806 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15.142 s
[INFO] Finished at: 2020-04-17T15:07:28+03:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.googlecode.maven-download-plugin:download-maven-plugin:1.4.2:wget (include-openrtb-proto) on project doubleclick-openrtb: Execution include-openrtb-proto of goal com.googlecode.maven-download-plugin:download-maven-plugin:1.4.2:wget failed: java.lang.ClassNotFoundException: com.googlecode.download.maven.plugin.internal.CachedFileEntry -> [Help 1]

Solution:
$ sudo rm -r ~/.m2/repository/.cache/download-maven-plugin/

Need to remove dependency on OpenCSV 2.3, which contains a couple of severe bugs. We're lucky that none of these bugs impact openrtb-doubleclick, but due to these bugs, dependencies of OpenCSV 2.3 are blocked in Google's internal projects. We could downgrade to an older version known to not have these bugs with small changes, but I'd rather not depend on this library anymore; version 2.3 is more than three years old so the project seems abandoned. Could shop for an alternative CSV library but it's a good opportunity to further reduce our dependencies, since our needs are very simple so a custom parser with only the basic features will be good enough.

Hi @opinali,

While using the DoubleClickOpenRtbMapper and testing it with Video requests, I notice that the mapper is not creating Video impressions.

I believe the reason is the following code:

    if (dcSlot.getWidthCount() == 1) {
      video.setW(dcSlot.getWidth(0));
      video.setH(dcSlot.getHeight(0));
    } else if (dcSlot.getWidthCount() > 1) {
      if (interstitial) {
        video.setW(dcSlot.getWidth(0));
        video.setH(dcSlot.getHeight(0));
      } else {
        logger.debug("Invalid Video, non-interstitial with multiple sizes");
        return null;
      }
    }

In that code above, the mapper is returning null when it is an interstitial request and has multiple-dimensions.

After enabling debug level in the mapper class, I could see several the messages:

...
[DEBUG] 2015-11-04 00:33:31,987 DoubleClickOpenRtbMapper - Invalid Video, non-interstitial with multiple sizes
 [DEBUG] 2015-11-04 00:33:31,988 DoubleClickOpenRtbMapper - Request has no impressions
 [DEBUG] 2015-11-04 00:33:31,988 DoubleClickOpenRtbMapper - Request has no impressions
 [DEBUG] 2015-11-04 00:33:34,561 DoubleClickOpenRtbMapper - Invalid Video, non-interstitial with multiple sizes
 [DEBUG] 2015-11-04 00:33:34,561 DoubleClickOpenRtbMapper - Invalid Video, non-interstitial with multiple sizes
 [DEBUG] 2015-11-04 00:33:34,561 DoubleClickOpenRtbMapper - Request has no impressions
 [DEBUG] 2015-11-04 00:33:34,561 DoubleClickOpenRtbMapper - Request has no impressions
...

So, for my surprise, it seems that AdX is only sending video requests to me that are both interstitial requests and has multiple dimensions - thus the mapper is always returning null and finishing the translation to OpenRtb with no impressions objects.

Wondering if this is the right approach I went to DoubleClick spec, where it can be read:

...
// For mobile interstitial ads (including ones where video ads are eligible)
// the first width height pair is the screen size (this is also the video
// player size for VAST video ads). Subsequent pairs are recommended
// interstitial ad sizes that satisfy the interstitial size restrictions,
// i.e. no bigger than the screen size and no smaller than 50% of width and
// 40% height.
repeated int32 width = 2;
repeated int32 height = 3;

So, from my understanding, it is possible to have video interstitial requests with multiple dimensions. In that case, we should get the first width and height as they should be the video player size.

Therefore, I would propose the following change:

    if (dcSlot.getWidthCount() > 0) {
      video.setW(dcSlot.getWidth(0));
      video.setH(dcSlot.getHeight(0));
    } else {
      logger.debug("Invalid Video, dimension of the video player not defined");
      return null;
    }

And no need for interstitial argument in the buildVideo method...

What do you think? Am I missing something?

PS: I did not open a PR as I read in the CONTRIBUTING file "...It's generally best to start by opening a new issue describing the bug or feature you're intending to fix..."

Hey guys,

I am using this library to convert protobuf bid request to openrtb request. However, switching to the latest version, I do not see DoubleClickOpenRtbMapper and other related mappers.

Can someone clarify why that is the case, and how to use the library without it.

Hi,

I tried following what you said in the wiki as we have our own implementation of the OpenRTB models:

Note: If adopting openrtb-core's RTB model classes is difficult for some reason, for example you have legacy code that uses another OpenRTB library, you can just add a step 4 that converts our OpenRtb.BidRequest to a JSON string that can be parsed back to the preferred OpenRTB model. See openrtb-core's documentation about JSON serialization. For the response just do the reverse, creating the JSON output and parsing that to our OpenRtb.BidResponse that can be mapped to DoubleClick. Notice that these additional JSON serialization steps are very fast (in the order of ~10 microseconds), so don't worry about extra cost.

My problem is that when I json encode the OpenRtb.BidRequest (using google GSON) most of the fields are suffixed with an underscore. For instance i have id_, imp_, imp_[0].banner_.btype_.

Our model respects the OpenRTB names (i.e. we have fields named id, imp, imp[0].banner.btype)
Is there anyway to correct this besides doing a complete mapping from one object to the other ?

Thanks

DoubleClickCrypto defines a ThreadLocalRandom member this way:

private final ThreadLocalRandom fastRandom = ThreadLocalRandom.current();

It uses this ThreadLocalRandom to initialize an initVector whenever we wish to encrypt data.
This permanently sets the fastRandom member to the creating thread's ThreadLocalRandom.
ThreadLocalRandom is, by itself, not a thread-safe object and all uses of it should be of the form:

ThreadLocalRandom.current().nextX(...)

This makes DoubleClickCrypto essentially not thread-safe.
Am I missing anything?

Hey,

thank you for all your work, could you please update the changelog for the newest release?