Detailed elaboration of our requirements:
1. i need a system to automatically identify market focus (i shall refer to them as topics - but pls distinguish them from topicmodel's 'topic') that are of concern in the financial market. ideally we should have NLP automatically find topics by count (i.e.: more count = more important), or align by time (i.e.: at a certain time, if the newsflow is increasing, AND price in the markets are volatile).
for now, if its difficult, we can hardcode a list of topics that we know are important now, for example, Brexit, Trade War, Fed Hike, US Recession, Global Slowdown, etc. these are "topics", also known as "market concern". this list is to be changed from time to time, so i need a design to explain how to do this in real time (i.e.: when extracting out the computed series for analysis, instead of during process time).
2. what i want is a system, which using these topics, can identify entities related to these topics.
that is because, i do not want to process the article's sentiment. this is to extracting the writer's sentiment. instead, i want a list of important people related to these topics. For example, Brexit have many people important to it (e.g.: Theresa May, Donald Tusk, etc, etc). i want the sentiment of each of this person.
3. in achieving the above, we need to have a way to:
a. identify how entity relate to topics?
b. how to use clustering so topics of the same "meaning" and entities of the same "person" can be grouped together?
c. once that, how to identify the sentiment of each of this clustered entities?
d. finally, how to group the right clustered entities into the right clustered topics? for example, Donald Trump may fall under Trade War, or Fed Hike. it is important to distinguish.
4. what i need is to design and build a system that does above 1-3, by first taking over the existing code, and implementing net new work.
5. once above are done, then we need to plot the series, and explore its trend against price. i suppose 1-3 is a very difficult task, so we require deep thoughts and proper thinking and design capabilities.
-------------------------------------------------------------------------------------------------------
right now, our architecture is:
a. crontab crawling and inserting news item to a mysql DB
b. we have a cluster of AWS spot instances which reads these DB items and in parallel process them using NLP, and save the results into mongo
c. we run a python plotting file to extract from mongo and achieve the overall results.
hence, you're required to be familiar with AWS instances.
-------------------------------------------------------------------------------------------------------
the project is ongoing and we require daily interactions with our team. we have worked with many freelancers and failed to establish a working momentum largely because the individuals are too microscopic in thinking and are unable to work independently. pls ensure you're able to have high work ethics otherwise we're just wasting our time.
About the recuiterMember since May 20, 2018 Mark Perlstein
from Mpumalanga, South Africa