counts data very different from yesterday about coronavirus-data HOT 32 CLOSED

Using the NYC data reported as of the Diagnosis-Date and using multiple Case-Hosp-Death.csv files to represent the Reporting date we can see the effect of back-dateing. We can see that the top of the bar should match the NYC Health dept single color bar chart. The changing colors below signify the ever changing story. In my case, I would not want to use case growth past 3/25 (maybe the 26h) for its predictive qualities; because of the ever changing story. I am looking at how much did the latest day (4/04) impact each of days in the past. Basically, this means that we are not getting the real story for 10 days, using only diagnosis date. (ouch)

I would like to point out something, that is the weekend effect. This is more prominent when viewing the New Cases via the Diagnosis Date. New Cases Dip over the weekend (21/22 & 28/29). My hypnosis is that the Health Care system is impacting the discovery rate, that is many doctors office's are only open durning the work week and sick appointments are pushed out into the coming week. The urgent care & hospitals would be in operation over the weekend. I think the Reporting Date steadiness is a reflection of the Lab's always having a backlog.

I know this issue is confusing, if not frustrating. This kind of issue can be difficult to properly research and more importantly properly present in an informative way. I hope this is helpful.

@iswagner @ryanwitt @Piste @braxtonmckee @brucegarro @joansobo @tandcsurf @speedplane @HarryBirtles @lavenj @brucegarro @ret394

from coronavirus-data.

Hello @mmontesanonyc could you please update testing.csv, I believe this is a useful file until the effects of the diminishing testing LAG time makes itself evident. I believe we need graphical showing Positive cases by two x-axis (Diagnosis-Date, Results-Date). Personally, I would love to know how many test are outstanding as-of-the diagnosis date. I believe Hospitalizations and Deaths columns should be ignored testing.csv because the have nothing to do with the Diagnosis-Date or the Response-Date.
This LAG issue will be a sore point, with all concerned stakeholders and may lead to distrust of the data until the LAG has fully worked out which may be after the APEX should have been determined. Right now it looks like the real story a week ago was worse than stated. A good visualization should explain the temporary effect of the data LAG.
NOTE: NYC is now rolling out the rapid test. New Test results will have same day results while we will be getting older results. The new view based on Diagnosis is the preferred view. We just need a simple view to explain LAG for the interim period.

from coronavirus-data.

Hello @themosta:
I am making a spit-ball projection base on all of 12 days of experience with regards to the New-Case data lag. I'm making an assumption that we are getting in 1.2% of the New Case as of the Reporting Date. I am making the an exception for the weekends, I am assuming that we are getting in 2.2% of the New Cases as of the reporting date.

My spit-ball is adjustments are based on

I used 1.2% as a compromise (or fudge) over day 10,11,12. My view is that processing is getting more automated and that we are getting more New Cases in the first day (doubled this week).
In any case it's all just a SWAG!

This is how the new cases are looking to me.

This is looking like 6k to 7k new cases a day in this paste week.

from coronavirus-data.

The PDFs are by reporting date, see here, while the GitHub data is by date of diagnosis, so presumably these are not necessarily the same date.

I assume reporting is waiting on confirmation by a test, while diagnosis may be done before this.

from coronavirus-data.

The City Health dept is now deferring us to the State. I did find the state dash board and with some work you can re-create the Positive/Negative results along with the cumulative Positive results for the 5 counties (boroughs) of NYC. The information is provided by TEST RESULT date (or REPORTING DATE). This is the format that we were used to seeing as of last week.
NYS dashboard is https://covid19tracker.health.ny.gov/views/NYS-COVID19-Tracker/NYSDOHCOVID-19Tracker-Map?%3Aembed=yes&%3Atoolbar=no

Note: The Testing.csv file did give us the ability to see the test results at the Diagnosis-Date (the date the test was ordered, most likely administered). Just for context there is ruffly a 3-10 delay between Diagnosis & Reporting. Reporting is just what we have been used to seeing and looks more consistent. Consistently up and to the right! (Source: NYS Dashboard download) This is the same chart as above without looking at the cumulative positive cases.

The continued rising POSITIVE NEW CASEs reported as of the Reporting Date is largely being pushed into the prior 3-10 days prior days. (Source: 6 version days of Case-Hosp-Death.csv) Data rejiggered by comparing files to determine level of backdating.

from coronavirus-data.

This is my LibreOffice-Calc (excel) version the multiple versions of the CASE-HOSP-DEATH.csv files comparing each day to document the level of back-dating (Lag). I have included an Instruction tab for those of you who are interested in building dashboards, so that you may figure out what kind of rejigger of the data that I am doing.

https://github.com/DTPOTO/DTPOTO_Repository/blob/master/NYC_New_Cases_by_Diagnosis_Date_vs_Reporting_Date_v1.ods

from coronavirus-data.

Yes, it looks like the new case numbers have been revised downward significantly. Do you know why this is @mmontesanonyc?

from coronavirus-data.

It's as if the two data sets are entirely different.

from coronavirus-data.

I noticed this too. But the nyc health site (screengrabbed below) says this is the "number of positive cases by diagnosis date". I think the PDFs were showing the number of positive cases that had been confirmed at the cuttoff time, where this data is aligned to the date when tests were administered. Given that there's been a substantial delay in getting tests back from labs it makes sense that the new curves we're seeing here are severely front-loaded vs the data we saw in the PDFs.

@mmontesanonyc - thanks for your work on this. Can you confirm or elaborate?

from coronavirus-data.

Thank you for your work on this @mmontesanonyc

from coronavirus-data.

If you look at the daily delta of total cases between the PDFs and the github data, you can see that the github data has the cases much earlier. It seems like @braxtonmckee has it right -- PDFs had totals as they were known at the time, and this data actually gets amended as new numbers come in

from coronavirus-data.

thank you @HarryBirtles. I noticed that today's PDF's no longer have the graph on the bottom showing the running total by day. Does anyone know if they plan to put those back?

from coronavirus-data.

The ReadMe fills comments on the Testing.csv file acknowledges there is a Test-Results Lag from the Diagnosis Date. Based on the comments in the ReadMe file, I am expecting to get multiple rows for a Diagnosis date for each Extract date.

From Readme in GitHub
testing.csv
This file includes counts of New York City residents with specimens collected for SARS-CoV-2 testing by day, the subsets who tested positive as confirmed COVID-19 cases, were ever hospitalized, and who died, as of the date of extraction from the NYC Health Department's disease surveillance database. For each date of extraction, results for all specimen collection dates are appended to the bottom of the dataset. Lags between specimen collection date and report dates of cases, hospitalizations, and deaths can be assessed by comparing counts for the same specimen collection date across multiple data extract dates.

from coronavirus-data.

Took some liberties by reworking the testing.csv file to calculate, Negative-Test, Positive-Test, Hospitalizations, and Deaths. Results can be seen
here
Assumptions:

1. Tested_Negative = Number_Tested - Number_Confirmed
2. Treated/Tracked_NonHospilized = Number_Confirmed - Number_Hospitalized
3. Hospitalized (Adj) = Number_Hospitalized - Deaths
   • This may be an errant assumption

Observation:

• Testing Protocol may have changed at the guidance of the CDC to conserve supplies, to Test only if Treatment Choice would be effected by Test-Results. as-of-3-21.
• Otherwise Test Results are lagging by as much as 10 Days.
  This would greatly complicate assessment (determining New-Case-Apex)

from coronavirus-data.

Took some liberties by reworking the testing.csv file to calculate, Negative-Test, Positive-Test, Hospitalizations, and Deaths. Results can be seen
here

Love your viz!

from coronavirus-data.

Hi @mmontesanonyc and @DTPOTO thank you for working on this. Would it be possible to also add the confirmed cases by reporting date (as was done in the PDFs) as well as by diagnosis date (as is done in Github now). I think both sets of data provide value, especially because so many were looking to the reporting date PDFs for the past few weeks.

from coronavirus-data.

Hi @DTPOTO . I was comparing the information between the data set for March 31st and April 1st, and the new case count, new hospitalized count, and death count changed drastically from one day to the other. New case count for example, rose by 1,235 just on cases before pre March 29th. In March 30th alone, there was an increase of 1,759, from 1241 to 3,000. In other words, new cases reported on April 1st were not the 953 shown in the March 31st date, but those 953 + the other 2,994 additional cases from previous dates, for a total of 3,947 new cases. Is this ok? I understand that, as new data comes in, previous datapoints can be updated so as to show the truth (as it is done when an economic datapoint is "revised" months afterwards). I just want to make sure I am following the correct information. Please find my calculations attached.
NYC Cases.xlsx

from coronavirus-data.

Hello @joansobo, I have looked at your spreadsheet. You have done a good "data sanity check". Just speaking about the POSITIVE TEST results. You have shown that there is an increase in Positive Test based on back-dating. That is the LAG between Diagnosis Date and Response date makes itself apparent. For example the increase in Positive test 2 days ago is 142%. We also have an impact as far back as 14 days ago. Using percentages and making the assumption that each day back would have similar impacts going forward. This is not a realistic asumption, I think this is a worst case assumption because of the new rapid test.
Click
NYC.Cases_Adj1.xlsx
to see how I adjusted you spreadsheet to try to anticipate the LAG effect. (Click Raw to download)

Click
NYC.Cases_Adj1_Graph.pdf
to see visual showing the test Response LAG.

I do expect this story will change, because the LAG effect should be diminished over the next 8 days.

Additional comment: I think the Hospitalization Dates may be effected by Multiple Hospitalizations by the same person and we are seeing the LAST hospitalization date. This would explain why the Hospitalization Counts are going down in the early days.

from coronavirus-data.

thanks for your response @DTPOTO. I agree with you in that some sort of adjustment should be put in place in order to be more accurate. The adjusment you suggest (which are based on real, observed data points) would be a lot more accurate if we used the data set of each day since this information started being gathered github. Yes, I agree this are worst case scenarios, however, we should do some adjustment unless the new rapid test changes the game. There are however many other variable that create lags (mostly communication related). I would assume the health system is not instantly connected, with a lot of manual bottlenecks.

from coronavirus-data.

echoing @DTPOTO and others, would be tremendously helpful to have the old pdf dataset visualized alongside the current csv. adding cases to the current graph as far back as 2 weeks really diminishes its usefulness until rapid testing is commonplace

from coronavirus-data.

@DTPOTO I have continued to review the information and there is something that caught my attention. So, we previously discussed that there are X amount of new cases reported on a new day, and those X new cases can be distributed backward into other days based on the day of the test. This was the conclusion we reached a few days ago, for lack of wanting to conclude that NYC personel in charge of publicating this information is reporting numbers in such a way so as to flatten to curve. So, giving the benefit of the doubt we have assumed that the new cases are being distributed according to the date of testing. There is one thing that did caught my attention, and I have not been able to figure out. Like cases, there are X amount of new deaths reported on a new day. Of this X amount of new deaths reported, there are many being reported backward as well, and not only for that new day. The only possible reasons for these is that a) this death has only come now to the attention of the authorities, the date of death had been wrongly recorded, or c) that deaths are being assigned to the day the person was tested. Neither b) or c) make sense. Only a) could be ok, but still does not sound reasonable. In my mind, if there is a new death, it can only fall in the day the person died, meaning there should not be any death being reported on days before the report date. Please check this out in the document attached.
NYC Cases.xlsx

from coronavirus-data.

Well stated "This was the conclusion we reached a few days ago, for lack of wanting to conclude that NYC personel in charge of publication this information is reporting numbers in such a way so as to flatten to curve." I am trying to stay away of think the system is consciously or worse yet unconsciously prep to data to hide from the truth. That being said you can accuse me being a Pollyanna but I am going to speculate on the Administrative Lag time between "Date-of-Death" and the "Reporting Date". This would be Kind-a-Like your option 'A'. The NYC-Dept-of-Health is a government agency getting information after the fact. If they are now trying to tell a story as it was last-week for positive test then it would make sense that they would do the same for Deaths (as well as Hospital Admissions). The readme documentation says that Deaths are as-of the "Date-of-Death". Presumably, we were getting the data as-of the "Reporting-Date". BUT, if that was the case WHY would we be getting soooo much back-dating every day! Additionally on the 4/01 some of the earlier deaths were eliminated... hummm the Walking-Dead, or reassigned to another county... maybe option-2. I am showing around 70% of the Deaths each day are being back-dated. This is a lot of continuous rejiggering of the data. If I assume that the last 3 days represented New Deaths then only around 25-40% were backdated. 25% late reporting in the time of a crises... maybe!

I am getting the same level back-dating with the New-Case adjustment based on Diagnosis-Date vs Reporting-Date. I am looking at the New-Case Chart and I agree that it looks like a flatting of the curve in the chart, but the informed viewer should know that you have to ignore what the chart is doing for the last 7 days. I don't know if the past 7-days will be rising or maintaining the flat perspective with a max of 3,500 new cases. I am actually stating to be concerned that the Flatting of the Curve is really a sign of a Health-System Capacity issue.

Again, the lack to transparency/communication on the Health Departments part is only playing into distrust of the data as well as their motivation.

@joansobo, I agree with your observations and your math.

FYI for ALL:
New York State is reporting Positive & Negative Cases as of Reporting date. If you pull just the 5 NYC counties the data matches the prior reporting using Reporting-Date.
https://covid19tracker.health.ny.gov/views/NYS-COVID19-Tracker/NYSDOHCOVID-19Tracker-Map?%3Aembed=yes&%3Atoolbar=no

from coronavirus-data.

Thanks for your last post, @DTPOTO, the backdating chart makes it very clear.

from coronavirus-data.

Hi @DTPOTO. I was trying to compare the numbers in the link you send us before. I will repeat this link below for quick of reference: https://covid19tracker.health.ny.gov/views/NYS-COVID19-Tracker/NYSDOHCOVID-19Tracker-Map?%3Aembed=yes&%3Atoolbar=no
How do you go about making the comparison? Maybe a single day example would help me out.

Please find attached the updated version of the file I have been using to track the progress of the virus. In here you will find a spreadsheet on reported deaths we have been discussing. Could you send me yours? My first read on the number of death had been that, the more the new deaths reported were backdated, the flatter the curve would look standing here today. But looking at the numbers now, I can only conclude the numbers are worse than expected for past weeks, and that I cannot know what is currently happening until 6 or 7 days from now. Please give me your read. I have been giving a lot more importance to number of deaths as that is what will give us a true signal as to how long this may last.

Anybody else is welcomed to help read the current situation.
NYC Cases.xlsx

from coronavirus-data.

@joansobo

1. https://data.ny.gov/browse?tags=covid-19
2. click the Title "New York State Testing"
   will take you to https://health.data.ny.gov/Health/New-York-State-Statewide-COVID-19-Testing/xdss-u53e
3. You could click on the the TRACK which is the link that I sent earlier
4. But to download click EXPORT to get the data file for all counties.

I have recreating the Testing/ spreadsheet from this data. (NEW CASES as of 4/06)
This is NEW CASES base on Reporting Date.
https://github.com/DTPOTO/DTPOTO_Repository/blob/master/New_York_State_Statewide_COVID-19_Testing%202020_04_06.xlsx
(Let me know if you have any problems working with this file. LibreOffice & GitHub is new to me)

As far as the Deaths, Yep I am seeing similar backdating because of Date-of-Death vs Reporting. The New York State Tracker does provide visuals on the Death counts my understand from any Issue chain that NYS is showing higher Death Counts in TOTAL for NYC. No one understands the descrpency at this time.

from coronavirus-data.

@DTPOTO I really like what you have done, looking at reporting lags- Does it seem to you that the reporting lags have decreased of late? (would be great if you have updated data)

have you tried to impute missing (not yet reported) cases using a "completion factor"?

from coronavirus-data.

kind of like this:

If you have 100 cases on day -3, and inflate it back by your historic calculated lag percent sequentially, would end up with 437 cases over the next 10-12 days. (multiplier is 4.4)
on day -4, would need to inflate by 3.3
on day -5 would inflate by 2.3, and so forth.

Now, I think the reporting lag has dropped significantly, so would want to apply recent lag data to get more accurate picture

Table 1

  | %Lag Today | Expected LAG |   |  
30 | 0% |   |   |  
29 | 0% |   |   |  
28 | 0% |   |   |  
27 | 0% |   |   |  
26 | 0% |   |   |  
25 | 18% |   |   |  
24 | 0% |   |   |  
23 | 0% |   |   |  
22 | 0% |   |   |  
21 | 0% |   |   |  
20 | 0% |   |   |  
19 | 0% |   |   |  
18 | 0% |   |   |  
17 | 0% |   |   |  
16 | 0% |   |   |  
15 | 0% |   |   |  
14 | 1% | 1% | 437.25 | 1.0
13 | 1% | 2% | 434.54 | 1.0
12 | 0% | 2% | 427.10 | 1.0
11 | 0% | 2% | 419.21 | 1.0
10 | 1% | 3% | 410.15 | 1.1
9 | 1% | 4% | 396.77 | 1.1
8 | 5% | 9% | 381.35 | 1.1
7 | 7% | 16% | 349.69 | 1.3
6 | 7% | 23% | 301.13 | 1.5
5 | 4% | 27% | 244.59 | 1.8
4 | 4% | 31% | 193.00 | 2.3
3 | 17% | 48% | 147.59 | 3.0
2 | 142% |   | 100.00 | 4.4

from coronavirus-data.

Hello @themosta:
I like the idea of using an experience factor to estimate the completion level for New Cases. Keep in mind that the completion level needs to be tempered by where we are on the curve. Gov Cuomo believes that we are near the APEX.

My expectation is that last week Diagnosis-Date, was going to complete between 4k and 4.5k based on what the trajectory of the New-Cases using Reporting-Date for the prediction of the Yet to be reported New-Cases.

My research using day over day New Case increase is showing a decreasing "Rate-of-Increase". So bottom line each week needs to expect less back-dating only because we should be anticipating fewer New-Cases. But that is belief base not fact based. Look at my comments in a closed Issue "Normalizing to 100,000". My original concern was invalid but I did document the basis estimating the tempering effect. This will be a big issue when anticipating actual decline in New-Cases.

With or with-out the tempering effect, I can certainly apply the experience factor that you suggesting. Good Idea.

from coronavirus-data.

@joansobo
This is the deaths with the backdating accounted for. There were a lot of backdated Deaths this morning going back at least 11 days (Dark Purple). This is pretty much that same thing that you were concluding a couple of days ago.

Additionally, @elash20 has been trying understand why NYS web site has about 1,000 additional deaths for NYC than what NYC is acknowledging. @elash20 heard this morning that Mayor Di Blasio announced that the City has only been reporting on Deaths within the Hospitals. Deaths at home have not been included. At sometime in the future we should see a 20% jump just to reflect with backdating of those deaths. Right now the 3,600 Deaths for the City is understated by apx 1,000.

This is what the Death backdating has looked like for the past week. I believe this all agrees with your spreadsheet. I just visualized things differently. (Big jump on 4/7 today).

I will be posting my spreadsheet in my repository this evening.

from coronavirus-data.

Nice work @DTPOTO !

I just tried compiling similar data lag stats over the last 2 days here: (easier if you download the PDF) https://github.com/briankoral713/NYC-COVID19-Data/blob/master/CV-19%20NYC%20Data%20Lag.pdf

from coronavirus-data.

Hello @ryanwitt, I don't blame you for being confused. The color coding is attempting to show where the backdating went to for a given Reporting Date (I forgot to put on the reporting date label). The Gray Patch Pattern is very Rough SWAG at projecting the backdating that will happen over the next two weeks (Expected-Lag). There are a number of reason why the Expected Lag may be wrong (it's just a spit ball).

1. It is using the 1st day Reporting which was 1% of New Case ultimately reported after 12 days for a Given Diagnosis date. But more recent reporting dates have been doubling the first day. Was this because of more effective Administrative or Just more New Cases overall. I chose to pick a more optimistic level of 1.2%. We will see over the next two weeks.
2. The second reason why any historical based prediction will be wrong is that we may have actually reached the APEX (in NYC). Looking at the pattern of New Cases change rate day over day over the past two weeks, we may have reach or at least close to the APEX. This observation is very much confounded by Easter Weekend, with potentially an artificial down tick that will have an offsetting artificial uptick over the next two days.

@themosta the above graph is what I was referring to when I said that the Historical biased prediction needs to be tempered by changing reality. Reaching the APEX is where the virus has run its course. (Note: the above graph is based on reporting date, not diagnosis-date)

from coronavirus-data.

@themosta
This is the New Case accounting for backdating from over the Easter Weekend. Saturday was definitely skewed, the first reporting was extremely low so the prediction is artificially low.

I have not attempted to impose any "tempering" based on either improved reporting or virus progression. But my feeling is that 7,000 will be the APEX. All bets are off should they start massive testing which will change the perspective but not the reality. Meaning that we will test more and show more cases, but we always had those additional cases. The current Testing Protocol is to Test only when Symptoms exist and treatment choice may be effected by the test results. This protocol is under-reporting the actual disease state. A testing protocol that includes random sampling in addition to the sick while useful to understand greater infection level will provide many more Positive test.

from coronavirus-data.