My Math Forum trendingBot, the trend finder

 Economics Economics Forum - Financial Mathematics, Econometrics, Operations Research, Mathematical Finance, Computational Finance

 January 30th, 2009, 02:20 AM #1 Newbie   Joined: Jan 2009 Posts: 8 Thanks: 0 trendingBot, the trend finder We (trendingIT) have developed an innovative algorithm facing data management in a different way: trendingBot, a powerful numerical simulation tool capable to find trends (equations describing relationships among the introduced data) with no user intervention (arbitrary user parameters). For further info, you can visit our webpage, our blog (for some theoretics) or take a look at some of our videos. We would love to discuss further about any issue. Thanks for your attention.
 January 30th, 2009, 09:25 AM #2 Global Moderator     Joined: Nov 2006 From: UTC -5 Posts: 16,046 Thanks: 938 Math Focus: Number theory, computational mathematics, combinatorics, FOM, symbolic logic, TCS, algorithms Re: trendingBot, the trend finder Actually, I find this very interesting. I thought it might just be random promotional spam, but it looks like a useful tool -- and one relevant to this forum (here in economics or in applied math). I must admit, though, a thousand euro is quite expensive. I started writing a piece of software to do something similar: curve fitting without user intervention. How does your system compare to others? I imagine it's much easier to use than R, and more powerful than the Excel Analysis toolkit, but how does it compare to (say) CurveExpert? I often find myself using some kind of trend finder when I have runtime data: running a program with input (size) x takes time y, for some reasonable collection of x and y values. How well does your program handle this?
 January 30th, 2009, 01:26 PM #3 Newbie   Joined: Jan 2009 Posts: 8 Thanks: 0 Re: trendingBot, the trend finder The main difference between trendingBot and conventional programs (based on conventional statistical methods) is the number of independent variables: trendingBot's algorithm can theoretically deal with a number as high as you want and in the current version, 10 independent variables!! We try to keep the algorithm unrevealed and I donīt want to say too much... it would be better to show its capabilities. You have various options: - Go to our webpage and accept the "blind test challenge": send as many data as you want (follow the instructions for protecting your privacy without losing information quality) and get completely for free and with no commitments the solution (trend, application range, expected error, etc.). - Ask for a trial version (from the webpage too). Note that you have to provide your address in order to send you the security device. If you are not very sure (because of the price, for example), just do the test: try a complex set of data (3 independent variables or more) with one of the programs that you propose and send it to us. You will see the solution and assess wheter it is worthy or not. I hope I have answered your question
February 1st, 2009, 01:46 PM   #4
Global Moderator

Joined: Nov 2006
From: UTC -5

Posts: 16,046
Thanks: 938

Math Focus: Number theory, computational mathematics, combinatorics, FOM, symbolic logic, TCS, algorithms
Re: trendingBot, the trend finder

Quote:
 Originally Posted by trendingIT The main difference between trendingBot and conventional programs (based on conventional statistical methods) is the number of independent variables: trendingBot's algorithm can theoretically deal with a number as high as you want and in the current version, 10 independent variables!!
Support for lots of variables is good. How well does it handle noisy data? I can imagine a lot of cases where there is a trend but it's hard to read under all the noise.

Quote:
 Originally Posted by trendingIT We try to keep the algorithm unrevealed and I donīt want to say too much... it would be better to show its capabilities.
Fair enough, I'll just pester you with questions instead.

How well does your algorithm handle nonpolynomial relationships? y = x^2.5, y = 2^x, y = x log x? How about exp((log x)^a) for noninteger a? (That last one is actually much more common than you'd expect.) Obviously recognizing those -- and worse, their multivariable versions -- is tough.

Quote:
 Originally Posted by trendingIT - Go to our webpage and accept the "blind test challenge": send as many data as you want (follow the instructions for protecting your privacy without losing information quality) and get completely for free and with no commitments the solution (trend, application range, expected error, etc.).
Sounds like fun, I might take it up.

February 2nd, 2009, 12:42 AM   #5
Newbie

Joined: Jan 2009

Posts: 8
Thanks: 0

Re: trendingBot, the trend finder

Quote:
 Support for lots of variables is good. How well does it handle noisy data? I can imagine a lot of cases where there is a trend but it's hard to read under all the noise.
It depends on how important noise is, there isn't any detailed study about this subject (just beginning, the exact applicability of the current version to any field of expertise has to be tested).
There is no "discrimination" part, right now trendingBot is powerful but dumb.
Our intention is including ASAP a new algorithm taking care of "multiple behaviours" (example: when for x=1 to x=3, y=x and for x>3 y=2.5x); logically, this part will take care of noise (by assigning a new equation or just by recognising and telling to the user).

Quote:
 Fair enough, I'll just pester you with questions instead. .
I am promoting a product and part of my work consist in being nice to pesters

Quote:
 How well does your algorithm handle nonpolynomial relationships? y = x^2.5, y = 2^x, y = x log x? How about exp((log x)^a) for noninteger a? (That last one is actually much more common than you'd expect.) Obviously recognizing those -- and worse, their multivariable versions -- is tough.
As said, the approach is different to the conventional ones, including the way a problem is faced: if you "provoke" a trend, mainly in case of being too complicate, it might not be found. The program (the algorithm) was created with an exclusive aim: helping to understand complex data sets, by giving a straightforward and immediate first approach. Once the raw data becomes difficult (let's say, more than 3 independent variables affecting a dependent one), the number of possible mathematical relations (trends) fitting within the trendingBot's definition (+-5% error with respect to the original data for, at least, 85% of the cases) is huge and the program is highly limited (combinatorics, even with the power of a nowadays computer drives to really long and time/CPU consuming calculations).

Conclusion: trendingBot offers ONE trend (the best one within the options it is considering) what, from our experience, gives more than enough certainty for NATURAL BEHAVIOURS. That is, if you introduce a set of data, suitable to be defined as "natural" (from experimental measurements of any kind, logical behaviours (economical/financial variables, human behaviours, highly conditioned situations (computers)), etc.) and trendingBot says "trend not found", you can be quite sure that there is no mathematical relation explaining them; on the other hand, if a trend is found, it wouldnīt for sure either the only one or the best one (out of all the possible mathematical relationships).

trendingBot is a sensible alternative constructed on the basis of a trial-and-error process: the taken path is not a completely comprehensive one, is the one that the experience has revealed to be the most adequate. Additionally, the whole process of algorithm improvement has just begun: nowadays it is the best option in comparison with other alternatives but it is just the first version of a different way to face the problem.

All the possibilities you propose are recognised as trends but under a different form (and worse, for example, showing a mean error (after being applied to the original data) of 1.3% instead the logical 0%).

Quote:
 Sounds like fun, I might take it up
This is the best way: firstly making sure that it actually works and only then trying to understand why (or buying it to reward our effort )

February 2nd, 2009, 05:30 AM   #6
Global Moderator

Joined: Nov 2006
From: UTC -5

Posts: 16,046
Thanks: 938

Math Focus: Number theory, computational mathematics, combinatorics, FOM, symbolic logic, TCS, algorithms
Re: trendingBot, the trend finder

Quote:
 Originally Posted by trendingIT As said, the approach is different to the conventional ones, including the way a problem is faced: if you "provoke" a trend, mainly in case of being too complicate, it might not be found.
Sure. Of course the first three examples I gave are actual growth rates for datasets I have, and there are programs that have runtime similar to the fourth (say, the NFS). So these aren't just random examples but the sort of data I would feed into a program.

Quote:
 Originally Posted by trendingIT Conclusion: trendingBot offers ONE trend (the best one within the options it is considering) what, from our experience, gives more than enough certainty for NATURAL BEHAVIOURS. That is, if you introduce a set of data, suitable to be defined as "natural" (from experimental measurements of any kind, logical behaviours (economical/financial variables, human behaviours, highly conditioned situations (computers)), etc.) and trendingBot says "trend not found", you can be quite sure that there is no mathematical relation explaining them; on the other hand, if a trend is found, it wouldnīt for sure either the only one or the best one (out of all the possible mathematical relationships).
That's a strong claim.

Quote:
 Originally Posted by trendingIT trendingBot is a sensible alternative constructed on the basis of a trial-and-error process: the taken path is not a completely comprehensive one, is the one that the experience has revealed to be the most adequate. Additionally, the whole process of algorithm improvement has just begun: nowadays it is the best option in comparison with other alternatives but it is just the first version of a different way to face the problem.
Sure, the problem is very difficult. The search space of possible formulas is large.

Quote:
 Originally Posted by trendingIT ANSWERING TO YOUR QUESTION All the possibilities you propose are recognised as trends but under a different form (and worse, for example, showing a mean error (after being applied to the original data) of 1.3% instead the logical 0%).
That's impressive.

February 2nd, 2009, 06:17 AM   #7
Newbie

Joined: Jan 2009

Posts: 8
Thanks: 0

Re: trendingBot, the trend finder

Quote:
 That's impressive.
I can probe it, just need some data (http://www.trendingit.com/src/blindtest_trendingbot.php).

 April 28th, 2009, 01:07 AM #8 Newbie   Joined: Jan 2009 Posts: 8 Thanks: 0 Re: trendingBot, the trend finder At trendingIT, we have changed our general policy and decided to share the main theoretical ideas behind "trendingBot". If you are interested on all this, you can go to: http://www.trendingit.com/src/trendi...theoretics.htm - there, you can read some quick-ideas documents about the program or more detailed ones analysing the current existing forecasting methods, program's pseudocode or its mathematical foundation ("comparison within an environment"). PS: CRGreathouse, thanks for your advice(s).
 April 29th, 2009, 05:31 AM #9 Global Moderator     Joined: Nov 2006 From: UTC -5 Posts: 16,046 Thanks: 938 Math Focus: Number theory, computational mathematics, combinatorics, FOM, symbolic logic, TCS, algorithms Re: trendingBot, the trend finder I like the trend! So, in your view, why is this methodology significant? How does it differ?
April 29th, 2009, 08:26 AM   #10
Newbie

Joined: Jan 2009

Posts: 8
Thanks: 0

Re: trendingBot, the trend finder

Quote:
 Originally Posted by CRGreathouse I like the trend! So, in your view, why is this methodology significant? How does it differ?
It relies on detailed combinatorics as a solution for the conventional-regression-methods restriction [3D -> 2 independent vs. 1 dependent] in a different way than usually (DOE or exahustive regressions).

trendingBot creates a fictitious variable (x1) [resulting from all the possible combinations among the independent variables, raised at different exponents] and performs polynomial regressions between it and the corresponding dependent variable - after trying all the possibilites (within the current capabilities of the combinatorics) and evaluating their performance, the best one is chosen (the one showing the lowest mean error after being applied to all the elements within the traning dataset). You can take a look at the pseudocode (http://www.trendingit.com/docs/en/tr...pseudocode.pdf).

main differences (what others do not offer):
- virtually, no user intervention (out of trend definition, but it is recommended to not modify the default values) -> just paste the values in.
- close to actual understanding -> high level of detail + in the safest-side-possible decision algorithm, reducing as much as possible an eventual overfitting.
-... in summary: offering the closest solution to regression analysis beyond 2 independent variables (actual version up to 10, but perfectly increasable).

but as said in some of my previous posts, the best way to know the difference is by actually testing it... Now it is easier: there is a lite version you can download whenever you want (it deals just with 5 independent variables but works forever) -> http://www.trendingit.com/downloads/...click.php?id=4.

 Tags finder, trend, trendingbot

,
,

,

# content

Click on a term to search for related topics.
 Thread Tools Display Modes Linear Mode

 Similar Threads Thread Thread Starter Forum Replies Last Post kumalh Probability and Statistics 3 August 2nd, 2011 11:02 PM stephanos Calculus 1 July 14th, 2010 03:45 PM jimminyjojo Number Theory 5 August 12th, 2009 08:14 PM trendingIT Linear Algebra 0 January 31st, 2009 02:17 AM trendingIT Advanced Statistics 0 January 30th, 2009 02:28 AM

 Contact - Home - Forums - Cryptocurrency Forum - Top