So the thing is, bitcoin mining isn't about solving mathematical problems, it's about doing random work for a reward. So the point isn't to solve a mathematical problem for the good of humanity, it's about finding the hash of a cryptokey... and the reason that you want to solve this is to prove that you did some work to get a bitcoin.
Here's the original bitcoin paper where the author describes the system.
http://bitcoin.org/bitcoin.pdf
See section #4.
I would recommend reading the paper - the beginning sections are mostly in understandable English. The only section which I struggled with was #11.
The part that doesn't entirely make sense is why people pay money for a bitcoin. So it makes sense to me that they are out there, but what makes someone want to pay actual money for them.
1. they are non-trivial to get and theoretically impossible to duplicate.
2. transactions are theoretically anonymous
3. they are not tied to any government and the total circulation is fixed so in theory they should be inflation-independent.
The main reasons for the value seem to be #2 and #3. There are some people who buy them for reason #2 who do not want to be tracked, such as conspiracy theory people, but also criminals and people laundering money. And then there are some people who buy them and hold them for #3 - like gold, they think that this finite resource will be a hedge against inflation.
Here's an article by The Economist which talks about Bitcoins and their value.
http://www.economist.com/node/21563752