Comments (9)
@boesing you are using YYYY
instead of yyyy
for your date parsing. These are two different things.
YYYY
represents "week of year", which is a tricky definition in the ISO year-week calendar. You almost never want this.
yyyy
represents the calendar year. What folks normally consider year.
FYI, I did your test with yyyy
and things were indexed just fine.
these dates are not integer, though elastic parses these dates to integer and I guess there is a bug.
We parse all values and index them into long
values representing the milliseconds since the epoch. So, even though you are not supplying a number, all dates are technically numbers and we store them as such.
from elasticsearch.
Pinging @elastic/es-search (Team:Search)
from elasticsearch.
Did you check the local time of the ES nodes ? My guess is , some of them having different timezones contributing to this issue of < 24 hr durations.
from elasticsearch.
I did not, as I can reproduce it in a single node already. even tho, the local timezone should be ignored since the timezone is required to be submitted in the documents value. thats why I explicitly disabled date_detection
in the mapping and defined the format of the date_range
to contain a timezone.
Therefore, I would not expect to have any sideeffects on es server timezone at all. But maybe my assumption is incorrect 🤷🏼♂️
from elasticsearch.
@boesing The timestamps in the exception are as expected. The min
value is indeed greater than the max
value, and this is not allowed when indexing in a date range field
min: 1704128073000
max: 1704124499000
Note the seventh digit in min
is an 8
, but its a 4
in max
Maybe I don't understand the issue?
from elasticsearch.
@benwtrent Could you check the data I am sending to elastic? these dates are not integer, though elastic parses these dates to integer and I guess there is a bug.
There is a difference in those dates round about 23 hours and a couple of minutes. Something within elasticsearch does not convert those strings into the correct integers and thus - ofc - the values in the error are reflecting an issue but that is not caused on client side.
if that would be a client issue, I do not understand why the same request works within elastic cloud which is on 8.11.0 with lucene 9.7.0.
from elasticsearch.
Ah, wasn't aware of that. So its the format which introduces the problem.
I will try that, thanks for clarification! That might fix our problem.
from elasticsearch.
@benwtrent What I still do not understand is, that if elasticsearch uses SimpleDate
as described here and here, how does that lead to issues for dates in April?
I've adapted our mapping but I still don't get the underlying issue with my concrete example.
- Why is it valid in elastic.cloud but not when running docker?
- How is the year of the week of week 18 not equal (both dates are in the same calendar week, but even if they weren't in the same week, the earlier date would be in a lower week)?
So I would expect issues end of december where the week of the year could become 2025 on december 30th, 2024 but not end of april 🤔
from elasticsearch.
Why is it valid in elastic.cloud but not when running docker?
I don't think it is? I got the exact same error as you when testing in cloud, however its obvious that the resulting min and max are incorrect for a range mapping.
How is the year of the week of week 18 not equal (both dates are in the same calendar week, but even if they weren't in the same week, the earlier date would be in a lower week)?
I honestly don't know.
But we are not using SimpleDate
. We do utilize the underlying Java date parsing, but it is much more complicated.
Here is the parsing once we have the temporal accessor:
Here is the code once we have parsed out the TemporalAccessor for WeekOfYear:
private static LocalDate localDateFromWeekBasedDate(TemporalAccessor accessor, Locale locale) {
WeekFields weekFields = WeekFields.of(locale);
if (accessor.isSupported(weekFields.weekOfWeekBasedYear())) {
return LocalDate.ofEpochDay(0)
.with(weekFields.weekBasedYear(), accessor.get(weekFields.weekBasedYear()))
.with(weekFields.weekOfWeekBasedYear(), accessor.get(weekFields.weekOfWeekBasedYear()))
.with(TemporalAdjusters.previousOrSame(weekFields.getFirstDayOfWeek()));
} else {
return LocalDate.ofEpochDay(0)
.with(weekFields.weekBasedYear(), accessor.get(weekFields.weekBasedYear()))
.with(TemporalAdjusters.previousOrSame(weekFields.getFirstDayOfWeek()));
}
}
from elasticsearch.
Related Issues (20)
- MV_APPEND tests failing on geo_shape HOT 2
- Need new filter should_not HOT 3
- [Discuss] Enable date detection for data stream naming scheme by default HOT 1
- Wildcard search case insensitivity on accented characters HOT 4
- LOOKUP ON expression HOT 3
- [CI] InferenceCrudIT testDeleteEndpointWhileReferencedByPipeline failing HOT 4
- LOOKUP shouldn't duplicate the output if the same field was already present in the input HOT 4
- [CI] DownsampleActionIT testRollupIndex failing HOT 2
- [CI] JdbcDocCsvSpecIT test {docs.testFilterToday} failing HOT 4
- [CI] Netty4TransportMultiPortIntegrationIT class failing HOT 4
- [ML] StartTrainedModelDeploymentAction does not return a response by the timeout value HOT 1
- [CI] MvAppendTests testEvaluateBlockWithoutNulls {TestCase=<cartesian_shape>, <cartesian_shape>} failing HOT 3
- Dynamic dimension update fails with binary hex string HOT 1
- Script score query fails with byte vectors encoded as hex string HOT 1
- Unexpected scores for uniform vector edge case when knn query is applied to`int8_hnsw` HOT 3
- [CI] RankFeatureShardPhaseTests testProcessFetch failing HOT 2
- Unable to correctly query data using 'query_string' HOT 2
- AccessDeniedException when starting up after update HOT 12
- Remove unnecessary boxed Double usage in SortingDigest HOT 1
- Consider forking to generic tp for every msearch request HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch.