samgozman / fin-thread Goto Github PK

https://medium.com/@yardenlaif/go-sync-or-go-home-errgroup-f91a0ee72d3f

https://pkg.go.dev/golang.org/x/sync/errgroup

It contains only a garbage, need to investigate this error and fix or delete this feature.

the idea is to be able to add custom data filer for each journalist. For example, only show news with numbers from TradingEconomics RSS provider. Or specific stocks

PreFilter works before OpenAI, PostFilter after

https://go.dev/blog/slog

They generate a lot of ads and bullshit news to the feed that is hard to filter.

It will just return the element from the page so it can be parsed by openai later

like news:market for troubleshooting

To omit only some fields

extra } sign in the end or ` mark. Should only return an array []

84badc84deff37534a92d542feb604d2 & 63ce96d721bc53be01e34fb824331f8c
9776da2fca32fe7dd2f8c4ee28b24156 & 608c24baa1d9994fdd65ae4637fb2ccf

Sentry Issue: FIN-THREAD-N

*composer.ErrCompose: [Summarise] error in OpenAiClient.CreateChatCompletion: Post "https://api.openai.com/v1/chat/completions": context deadline exceeded
  File "/app/jobs/summary.go", line 109, in (*App).start.(*SummaryJob).Run.func19
...
(1 additional frame(s) were not displayed)

Sometimes publication duplication occurs with same hash id, but different text. Most likely because it was fetched by different feeds, which is odd because there is a duplication check..

For example: 56aeda3ec56755b129d3cd42acde6213 and 54c53458fa323da8bd54e56e89a21655

Maybe it's a deployment issue. Need to save news in DB before publication and update it after (this will raise duplication error quicker)

Sentry Issue: FIN-THREAD-W

*errors.errorString: unknown currency: ALL
*fmt.wrapError: [job-calendar] Error fetching events: unknown currency: ALL
  File "/app/jobs/calendar.go", line 78, in (*App).start.(*CalendarJob).RunWeeklyCalendarJob.func13.1
  File "/app/jobs/calendar.go", line 48, in (*App).start.(*CalendarJob).RunWeeklyCalendarJob.func13
...
(3 additional frame(s) were not displayed)

To fix weekly calendar job

OpenAI generates lots of hallucinated hashtags, they are not production ready and should be used only internally. Examples: a0ed61227f4e4729ddba423208175b47 & 731fcf21192498dbef522918f8fda688

Apply best practices from https://docs.sentry.io/platforms/go/usage/concurrency/

News are not getting any updates, this method is not working.

Because the target is US market and large EU corpos

Sentry Issue: FIN-THREAD-9

*errors.errorString: [Job.saveNews][News.Create]: url is too long
  File "/app/job.go", line 128, in (*App).start.(*Job).Run.func2
...
(2 additional frame(s) were not displayed)

From 15 to 20 seconds.

Like https://www.together.ai/ or https://www.anyscale.com/endpoints, because llama and mistral ai can cost 10x cheaper

LLama 70b chat was really good in filtering news with this prompt, but it responded with text also. But with good explanation..

You will be given a JSON array of financial news. You need to remove from array blank, purposeless, clickbait, advertising or non-financial news. Most  important news right know is inflation, interest rates, war, elections, crisis, unemployment index etc.
Return the response in the same JSON format. If none of the news are important, return empty array [].

[{"ID":"1","Title":"This key element of ESG investing could drive real change at companies and boost returns","Description":"As environment and board diversity initiatives have gained broad acceptance, some are now trying to clarify the case for the middle pillar of ESG.","Link":"https://www.cnbc.com/","Date":"2023-11-19T16:48:26.569248+04:00","ProviderName":"cnbc"},{"ID":"2","Title":"Goldman Sachs analysts have increased conviction in these 5 stocks","Description":"Goldman Sachs analysts named a slew of stocks to buy heading into the holiday season.","Link":"https://www.cnbc.com/","Date":"2023-11-19T16:48:26.569248+04:00","ProviderName":"cnbc"},{"ID":"3","Title":"Ray Dalio says U.S. reaching an inflection point where the debt problem quickly gets even worse","Description":"Soaring U.S. government debt is reaching a point where it will begin creating larger problems, the hedge fund titan said Friday.","Link":"https://www.cnbc.com/","Date":"2023-11-19T16:48:26.569248+04:00","ProviderName":"cnbc"},{"ID":"4","Title":"The market thinks the Fed is going to start cutting rates aggressively. Investors could be in for a letdown","Description":"Markets may be at least a tad optimistic, particularly considering the cautious approach central bank officials have taken.","Link":"https://www.cnbc.com/","Date":"2023-11-19T16:48:26.569248+04:00","ProviderName":"cnbc"},{"ID":"5","Title":"Wholesale prices fell 0.5% in October for biggest monthly drop since April 2020","Description":"Wholesale prices fell 0.5% in October for biggest monthly drop since April 2020","Link":"https://www.cnbc.com/","Date":"2023-11-19T16:48:26.569248+04:00","ProviderName":"cnbc"}]

Multiple sections:

1. All previous events
2. Top news summary
3. Expected todays events

run summary before market opens.

Add links to the discussed news (use publication id)

var (
ModuleErrname = errors.New()
)

Replace strings like "read more" or replace links in regexp format (http/https)

Since they do not post a lot

To store how many news fetched, how many published at the selected date. And for future statistics data

Stocks news publishing should work like this: if stocks in the news is in our DB - post right away. Others should be posted with delay (schedule) or not posted at all.

Create a bot that will show new news and post them only after approval. For instance, news from some sources or news without stock ids etc.

Some tickers can be found in the text like (NYSE: STOCKNAME), $TLT etc. This could be used further to filter news before sending them to the openAi

It's only capable for tickers data, news is very restricted

One more probable compose fix, if escalates.

Sentry Issue: FIN-THREAD-8

*errors.errorString: [Job.composeNews][composer.Compose]: [Compose] error in json.Unmarshal: invalid character '"' after object key:value pair for object: [{"id":"98541d11313e9975ce3a9fed62abcd04","text":"China's Consumer Price Index decreased by 0.50 percent in November 2023 compared to the previous month." "tickers":[], "markets":["FXI"], "hashtags":["inflation"]
  File "/app/job.go", line 121, in (*App).start.(*Job).Run.func3
...
(2 additional frame(s) were not displayed)

Same news on one source could be in multiple feeds

Sentry Issue: FIN-THREAD-2

*errors.joinError: Provider benzinga:buybacks error: Failed to detect feed type
  File "/app/app.go", line 46, in main.(*App).CreateMarketNewsJob.func1
...
(2 additional frame(s) were not displayed)

For example "Sign up" or "to buy now", "climate change", "activists", "new study", "advice", "covid-19", "a study released", any links "*.com", "humanitarian", "award", etc.

2023/12/22 17:53:39 WARN [Run.BroadNews][publish] error="[Job.publish][json.Unmarshal] meta: unexpected end of JSON input. Value: "

Sentry Issue: FIN-THREAD-D

*errors.errorString: [Job.publish][json.Unmarshal] meta: unexpected end of JSON input. Value: 
  File "/app/jobs/job.go", line 210, in (*App).start.(*Job).Run.func7
...
(1 additional frame(s) were not displayed)

Sentry Issue: FIN-THREAD-3

*pgconn.PgError: ERROR: duplicate key value violates unique constraint "idx_news_hash" (SQLSTATE 23505)
  File "/app/app.go", line 65, in main.(*App).CreateMarketNewsJob.func1
...
(2 additional frame(s) were not displayed)

https://docs.sentry.io/platforms/go/enriching-events/breadcrumbs/

https://docs.sentry.io/product/issues/issue-details/breadcrumbs/

To many dashed between words is really a popular error of gpt. "to-be-known"

"hyphenation" or "hyphenated compound words."

validations, publishers and other stuff should be moved to other methods and chained to the struct. Run() will only have if checks to run specific job or not.
Other features:

• Should have OmitSuspicious() class to not to publish news with this flag on the early stages (but save them for analysis in some way)
• Should have OmitEmptyMeta() class to not to publish news with empty meta (after composer event happens). But still save them for analysis.

Maybe there is no need to store all omitted news data in DB, so maybe store only link to it and meta.

Parse U.S. economic calendar with forecasts and median values. Fore example - https://www.marketwatch.com/economy-politics/calendar?mod=u.s.-economic-calendar (but it's not very useful because of the delay, actual value will be displayed later)

There are lot's of them. Find something reliable.

1. Parse todays calendar plan
2. Check for changes starting from the planning time of the report until it will be fetched.
3. Create automatic posts with it

https://www.mql5.com/en/economic-calendar

https://github.com/Jeffail/gabs

And publish them with priority