Cross posting - Here is a quote from Coins101 in BTCT (pg. 164: March 2, 2016) with an attached document that is a work in progress

"For my day job, I actually deal with big data on pretty much a daily basis. I also have to stand up in front of senior executives and explain to them why stuff costs so much and why they need to increase a budget to get better quality data and data analysis tools.  I put the attached together initially for my benefit (I sort of need this format to think things through as though I am about to stand up in front of people to explain stuff) and then for others once some of the details were worked out....so, peer into my head to see work in progress:"

https://docs.google.com/presentation/d/1qyExM5c1Ws7eUEj3J_gJ76zFe2WQaLdqJHMbJIJ_jng/edit?usp=sharing

Sup Folks,

After having a few conversations last night I began to realize that trying to facilitate a conversation about "big data methodology" may be easier if it were known precisely what that is.

Quote from: rhinomonkey on March 10, 2016, 03:50:46 pm

Sup Folks,

After having a few conversations last night I began to realize that trying to facilitate a conversation about "big data methodology" may be easier if it were known precisely what that is. It is a rather vague combination of two buzzwords to people who don't know specifically what it means. So I thought I would break it down into some smaller questions (though not exhaustive) that we would like to focus on. I will definitely make different subtopics with these specific questions later on but I figured getting started with a few here would not hurt.

"Big data methodology" as I understand it, involves a few separate facets that need to be explored. These questions have been discussed to some degree by Georgem and Coins101. But to make this more accessible I have enumerated a few questions below.

What information do we gather?

What data is valuable to end users?

How do we ensure the accuracy of said data collection?

And most importantly: How do we gather information in a way that respects privacy and the principles of decentralization?

More on this later. Please feel free to add some questions that are important to this topic that I have not listed.

One question that I have is centered around the viability of collecting "Big Data" from bitcoin transactions.  Since bitcoin is a digital currency it can be spent from any location to purchase goods at any other location.  Thus how does knowing the location of the user bring any added value?  Brick and mortar establishments cannot then cater to the goods that an individual is purchasing online, because the very nature of doing business online cuts out the middlemen.  Targeted advertising in this vein does not seem to make sense. 

Perhaps meta advertising, such as campaigns for a specific region might make sense, but the idea that individual store locations are going to greatly benefit from the data sounds dubious.  People using bitcoin as a mode of payments are leaps and bounds ahead of the typical grocery/retail goods shopper and thus it might not be a great inference to assume they are doing primarily online shopping, which in turn means that the marketing data would need to be utilized in an online space and not by companies expecting paying customers to walk into their store.

If these assumptions and inferences have any merit it would seem that the data we can collect would be most useful to companies that have large online presence and advertising money to spend on internet campaigns.  I am curious what you think about the more targeted approach coins mentioned in a previous post.

Quote from: rhinomonkey on March 10, 2016, 03:50:46 pm

Sup Folks,

After having a few conversations last night I began to realize that trying to facilitate a conversation about "big data methodology" may be easier if it were known precisely what that is.

Yes exactly.

The thing is, the way coins101/stonehedge define "big data" in their proposal is already very oriented towards fin-tech specifically, but "big data" is a much more general term.

For example, search engines (like google) are representing "big data" in a much more general way.
And let's not forget that one main "field of interest" that has motivated me since I joined spreadcoin, is the creation of a decentralized searchengine.



So, when I say "big data" I always also mean a new kind of decentralized crypto-focused search engine, that will run on a network of servicenodes.

Just keep that in mind when talking about "big data". This term is HUGE!

So basically, you have "big data" wherever you ...

a) ...get access to enormous amounts of big data,
b) ...have a digestion process where those large amounts of data are prepared for user consumption.

Quote from: rhinomonkey on March 11, 2016, 09:42:15 pm

Georgem, if Big Data is that broad, then the methodology doesn't only have to be through bitcoin transactions correct??

Also, what in particular, from a coding perspective is possible? Like where at the most basic level will service nodes be able to pull data from?

Yes, the most atomic thing in a cryptocurrency is a transaction.

But basically the whole internet is where we can pull our data from.

More specifically: network traffic and blockchains. But we could also scan websites for interesting data.

More generally: everything we already have access to (whatever can be openly accessed on the internet), plus maybe also what other people are willing to give us voluntarily (coins101's geolocation GPS data falls in that category)

From a coding perspective very much is possible.

To just gather/record data is easy.

To guarantee data integrity over the whole servicenodes network is much harder.

Plus once we have a steady data stream, how do we digest/reduce it to something we can offer back as a service?

The endresult must always be a digest (something smaller deduced from the giant dataset we dissected), so that it can be queried in a searchrequest and given back as a result.