And hey, who needs decompression anywayz? Like those people who’ve had themselves cryogenically frozen, you could compress EVERYTHING and then wait for some revolution in discrete maths or something, so that you can decompress the information back to normal. At least your HDD wouldn’t be full in the meantime, eh?

n!/(Π:l∈L.nl!)

Using your first set of numbers for your lorem ipsum without punctuation, we can calculate that in bc:

define f(x) {
if (x > 1) {
return x * (f(x-1))
}
return (1)
}
f(457) / ( f(24) *f(4) *f(20) *f(12) *f(51) *f(4) *f(6) *f(2) *f(38) *f(27) *f(16) *f(16) *f(15) *f(9) *f(4) *f(19) *f(32) *f(32) *f(35) *f(5) )

This comes out as the rather scary looking number:

24979676534385811028057980738663929391311867252395764304516513933684\
07606995096461303297817015351090284443259502380977579433712838416708\
45657900380181004097489062467705784954149469973163067321876126281229\
95907974733847026378002417559136955864960723139702457366278743422910\
76636180848502418461402452003145682339219616138491354732354517049290\
51034504318969704763573143447683688097296432449198854267451150579323\
49228888314823997292233314267179896978363474923474304843679668468255\
73366601896030037525678558877315402684142226202299224543792913670945\
70000940885751377191355034261912546595703027389104226535408140288000\
000000000000000000000000000000000000

…which is rather a lot of permutations.

If we assume 256 bits of hash data, we can divide this by 256 to get the number of permutations which will have the ‘correct’ checksum:

21572869700269331363124886806112228769291733255504027739303788591328\
37593298077865107496638668440706109266915612818099283527034418172707\
01882144552441912911611801039139944057017098915043865571617967016035\
92589594716339665906531835614143615869909227133379553874233921484767\
47750403478423115256020571862056979617852935949009025260523624674180\
80639965507183997390890815727819484329822253815589392749146335206762\
86330737257959530160336903679419768794361552184405895656881407312010\
61956861464362784040149412132287866816164180237900313291906666105277\
855060107509415077378619659

That gives us one ‘true’ positive, and 21572869700269331363124886806112228769291733255504027739303788591328\
37593298077865107496638668440706109266915612818099283527034418172707\
01882144552441912911611801039139944057017098915043865571617967016035\
92589594716339665906531835614143615869909227133379553874233921484767\
47750403478423115256020571862056979617852935949009025260523624674180\
80639965507183997390890815727819484329822253815589392749146335206762\
86330737257959530160336903679419768794361552184405895656881407312010\
61956861464362784040149412132287866816164180237900313291906666105277\
855060107509415077378619658 false positives.

Using a dictionary is a fairly smart way of eliminating a lot of that, which reduces the order of the problem to something close to putting the correct words in the right order, but I don’t know if dictionaries are allowed in the competition’s rules? If they are I’d imagine most approaches exploit a dictionary in some way.

This drastically reduces the number of bruteforce possibilities, but I
don’t really know how to express this mathematically – I believe
‘permutations’ is the mathematical field I’m looking at, but it’s well
beyond the calculus level I stopped at, I think.

The number I was thinking you meant was the number of permutations after considering your letter set, which is actually

n!/(Π:l&memb;L.nl!)

(and if html doesn’t have capital π or set membership entities, I shall be upset)

where L is the set of letters in your dictionary and nl is the number of instances of the letter l in the message.

I don’t know how big that is for your 457 character message, but it’s probably quite big, although it should be significantly smaller than 29^457.

“Gwyn came up with the suggestions of using two hashes – if you hash the final file with, say, MD5 and SHA1, it becomes extremely unlikely that a solution which matches with both hashes would be a false positive.”

What’s true to say is that it’s less likely a double-match would be a false positive than a single match is. If the second hash is, say, 128 bits, then it’s 2^128 times less likely that it’s a false positive.

Unfortunately, when the message length (in terms of the amount of information in the message) is so much longer than the hash, the chances of any given positive result being the correct result are are so much smaller than the chance of a it being a hash collision, that there are still far more double-hash collisions than there are genuine positives.

It’s just about plausible for the first number you mention, 457^29 case, which is about the same as 2^256, (which is to say, there’s just as much information in the hash as there is in the ) assuming you have a total of 256 bits worth of hash, but it doesn’t scale up: the probability (and therefore number) of false positives increases exponentially with the amount of information you add, or linearly with the number of permutations.

To break it down to a trivial case to make it obvious, say you have some letters you’re trying to arrange in such a way that there’s only 256 combinations of them. This naturally takes up 8 bits, and you try to encode this using a 2-bit hash. No matter how good, algorithmically, that 2-bit hash is, it can only ever reduce the number of possibilities by a factor of 22, so you’ll still have 256/4 = 64 positives, 63 of which are false. The same things applies when you scale up to 128-bit hashes with 2^(128+8-2)-bit messages.

Wowwie, I’m such a killjoy. But at least it’s Friday, right?

You argue that with a 29 letter alphabet, you have 457^29 possible 457-character strings, of which only one is your lorem ipsum example. This is true, but ignores your compressed data entirely: for instance, one of those strings is 457 z’s, which you can clearly eliminate from consideration by comparing to your a4530 b660 c2406.. data. In fact, taking this approach you’d have an even shorter compression algorithm: just tell someone the message length and the hash, and let them generate strings of that length and test for collision to recover the message.

By specifying the aggregate data, you’ve already told me what 457 letters I need, just not the order; so I have a pool to draw from. Were all those letters distinct, I’d have a 1/457 chance of getting the first one right, then 1/456 of the next and so on… for 457! possibilities. Still awful, but better- and it would also means that capitals or other symbols make no difference to the odds of success- imagine pulling unique scrabble tiles out of a bag and lining them up; whether the string is right or wrong depends on the order out the bag, not the alphabet employed to make the tiles.

However, as we don’t have 457 distinct tiles, but some selection of 457 from 29 (or 55, or whatever) types, a lot of the strings generated will be the same. For instance, v1w1×1y1z1 is 5 letters, with 5!=120 possible arrangements. But a3b2 only gives you 10 possible strings- choose 2 places out of 5 to put the b’s, and the a’s fill in the rest. Working out the possible unique arrangements of 29 symbols with multiplicity sounds a bit tough for a Friday morning, but I’ll keep it in mind if I’m teaching Discrete Math again next semester!