Case sensitive skill/spell lookups

Posted by Samson on Sun 28 Jan 2007 01:27 AM — 32 posts, 118,856 views.

USA #0
An interesting problem has developed at some point with the current SmaugFUSS. It seems that skill/spell lookups for pretty much any purpose have turned case sensitive and we can't find the reason for why.

The lookup problem appears to be at lest somewhat related to the various bsearch_skill_* functions. In trying to track down the cause, I find myself confused and wondering just what these things do.


int bsearch_skill_exact( const char *name, int first, int top )
{
   int sn;

   for( ;; )
   {
      sn = ( first + top ) >> 1;

      if( !IS_VALID_SN( sn ) )
         return -1;
      if( !str_cmp( name, skill_table[sn]->name ) )
         return sn;
      if( first >= top )
         return -1;
      if( strcmp( name, skill_table[sn]->name ) < 1 )
         top = sn - 1;
      else
         first = sn + 1;
   }
}


All that looks like for the most part is a cryptic mess that someone wrote to make it look like they were really smart or something :P

It sure seems to me that if all this is doing is looking for a name match against a certain skill name that one could just blast their way through the skill array looking for it rather than doing all this obfuscated mess.

Could someone who knows what this does please explain it in a way that most of us will understand? And perhaps suggest an alternative method to do this that's easier to figure out?
Australia Forum Administrator #1
Judging by the name "bsearch" it is doing a binary search. This is more efficient for large tables, as the search time is proportional to the log of the number of items in the table, rather than the actual number.

However on fast machines, the difference may be a bit academic (for 100 skills or so).

Basically it splits the table into 2 (say for 100 items, it looks at position 50). Then if the desired item is below 50 it looks between item 1 and 50, otherwise 51 to 100.

By repeating this operation, it can narrow down the comparisons by halving the search space on each iteration.

So, say the wanted item was at 99 (in a table of 100), you would quickly get there like this:

  • test item 50 (greater) (increment is 50)
  • test item 75 (greater) (increment is 25)
  • test item 87 (greater) (increment is 12)
  • test item 93 (greater) (increment is 6)
  • test item 96 (greater) (increment is 3)
  • test item 98 (greater) (increment is 2)
  • test item 99 (greater) (increment is 1)


This would be the worst-case scenario for a linear search, but it has found it in 7 steps.

In your case, possibly making the test not case-sensitive would solve the problem, but you must make sure they are loaded in correctly (that is, in not case-sensitive order).

Binary searches only work if the table is in sequence.
USA #2

/*
 * Lookup a skill by name.
 */
int skill_lookup( const char *name )
{
   int sn;

   if( ( sn = bsearch_skill_exact( name, gsn_first_spell, gsn_first_skill - 1 ) ) == -1 )
      if( ( sn = bsearch_skill_exact( name, gsn_first_skill, gsn_first_combat - 1 ) ) == -1 )
         if( ( sn = bsearch_skill_exact( name, gsn_first_combat, gsn_first_tongue - 1 ) ) == -1 )
            if( ( sn = bsearch_skill_exact( name, gsn_first_tongue, gsn_first_ability - 1 ) ) == -1 )
               if( ( sn = bsearch_skill_exact( name, gsn_first_ability, gsn_first_lore - 1 ) ) == -1 )
                  if( ( sn = bsearch_skill_exact( name, gsn_first_lore, gsn_top_sn - 1 ) ) == -1 )
                     if( ( sn = bsearch_skill_prefix( name, gsn_first_spell, gsn_first_skill - 1 ) ) == -1 )
                        if( ( sn = bsearch_skill_prefix( name, gsn_first_skill, gsn_first_combat - 1 ) ) == -1 )
                           if( ( sn = bsearch_skill_prefix( name, gsn_first_combat, gsn_first_tongue - 1 ) ) == -1 )
                              if( ( sn = bsearch_skill_prefix( name, gsn_first_tongue, gsn_first_ability - 1 ) ) == -1 )
                                 if( ( sn = bsearch_skill_prefix( name, gsn_first_ability, gsn_first_lore - 1 ) ) == -1 )
                                    if( ( sn = bsearch_skill_prefix( name, gsn_first_lore, gsn_top_sn - 1 ) ) == -1
                                        && gsn_top_sn < top_sn )
                                    {
                                       for( sn = gsn_top_sn; sn < top_sn; ++sn )
                                       {
                                          if( !skill_table[sn] || !skill_table[sn]->name )
                                             return -1;
                                          if( LOWER( name[0] ) == LOWER( skill_table[sn]->name[0] )
                                              && !str_prefix( name, skill_table[sn]->name ) )
                                             return sn;
                                       }
                                       return -1;
                                    }
   return sn;
}


Ok. So given the bsearch_skill_exact function I posted before, and this skill_lookup function taken from Smaug, can you tell us what the purpose is? I'm assuming all it wants to do is find a skill in the table by its name, correct?

I'm not sure I grasp this properly, but it seems to me it would be faster to just run the table than perform 12 separate binary searches. Perhaps there is a better way this code can be written and still maintain the efficiency desired?
Australia Forum Administrator #3
Yes, well doing it 12 times seems to me to be throwing away the efficiency the binary search gives you.

Probably a simple linear search would be just as quick in this case.

Also, if you are using C++ by now, then you could make it into an STL map, which permits a direct lookup.
USA #4
Nah, this was about SmaugFUSS, it's still C, so can't use the STL stuff.
USA #5
(Edit note: fixed various small things.)

I believe the intention of nesting the binary searches like that is to search first in the most likely candidate skill ranges. Unfortunately, that intention is misguided when you are doing binary search, and falls right into Knuth's saying: "premature optimization is the root of all evils".

Binary search is an O(log_2 n) operation. What that means is that the time it will take to search the array is proportional to the 2-logarithm of the size of the array, modulo some constant factors.
Quote:
Aside:
Big-O notation is a way of measuring algorithm complexity by saying what function the time will be proportional. O(n) means that the function duration increases linearly as the input increases; O(1) means that the algorithm always takes the same time no matter how big the input is (e.g., deleting the first element of a linked list). And so forth. It's a very powerful way of thinking about how an algorithm will worsen with respect to time as your input size grows.

Let's look at some case studies:

Array of size 16: binary search will need ~log_2(16) = ~4 lookups to find the element.

Array of size 64: ~log_2(64) = ~6 lookups.

Array of size 128: ~7 lookups.

Array of size 256: ~8 lookups.

Array of size 1024: ~10 lookups.

Array of size 65536: ~16 lookups.

As you can see, the time it takes to look something up increases very slowly even as the input array is multiplied by factors of 2.

(You can also see that for Nick's example of size 100, we needed 7 lookups which is consistent with the numbers above.)

Therefore, dividing the binary search into several segments shows a lack of understanding of how binary search behaves. There's really no need to try to search the most likely segment before others, unless (a) the segments are huge and (b) the probabilities of being in the various segments are _very_ disparate.
Quote:
Aside:
Consider a case where you have an array that can be divided into two sub-categories, and you know that a search will find a result from either one or the other. Further imagine that you know (somehow) that a request has a 99% chance of being in one segment and 1% chance of being in the other. Further assume that searching the arrays will take you a fair amount of time, perhaps because the arrays are very, very large (several million elements). And finally, perhaps the binary search itself is expensive, involving complex comparisons at every step of the way.
In a case like this, it might be worth your time to divide the search, trying first in the more likely segment.
But even so, you shouldn't go to this trouble until you know concretely that you are paying for this. You shouldn't go out of your way to optimize something without actually timing it first. It might turn out that improving your "really expensive search" will in fact only gain you a few milliseconds per second. (That is, you only gain a few nanoseconds per individual call to the function, and results only become "significant" when you sample over a large duration of time, like a second.)

The algorithm is an iterative version of binary search. It's a little less intuitive than the recursive version (to me, at least) but it works. The original author was probably trying to avoid the "overhead" (gasp) of a function call. If the recursion depth were to be to several hundred deep, it would start mattering (sort of... to be frank, even then, not really, because in the bigger picture this function is called relatively rarely); but in this situation the difference is basically noise.

This line:
sn = ( first + top ) >> 1;
is a "clever" way of dividing by two. Shifting right by 1 bit is the same thing as an integer divide by 2. For example:
101 binary is 5 decimal.
Shift right: 10; that's 2 decimal.

(Conversely, shifting left is the same as multiplying by 2.)

Shifting is indeed more efficient than actual division. Even a modestly intelligent compiler, however, knows that dividing by 2 is the same thing as a shift, and will optimize that all on its own if you have optimization turned on.

The point is that it's not useful or time-efficient to killer-optimize something that won't be called very often. Something called even 50 times a second that is basically fast doesn't need to have every last assembly instruction squeezed out of it. Now, if you had a loop that ran, say, 100 times a second, with lots of sub-loops -- as is commonly seen in 3d games -- it would matter. But in this case, it really falls into Knuth's expression.


Now, on to your problem.

This line:
if( strcmp( name, skill_table[sn]->name ) < 1 )
is using an exact string comparison.
I don't know if this is exactly the source of your problem, but what that means is that it chooses the next sub-area of the array to search based on a case-sensitive comparison. That can spell trouble because A < b but b > a. I would suggest using str_cmp.

I would also suggest caching the value of str_cmp by storing it into an integer after the first calculation. That way, if you find that you need it again, you don't need to scan the string again; you already have the comparison result.

But be careful about Nick's point; if you use case-insensitive searching you *must* make sure that the values are loaded (or later sorted) into case-insensitive order.

Also note:
Some of their checks are exact, some of them are prefix-based. It looks like it does everything exactly first, and then does everything by prefix. The best fix would be to have two binary searches; one exact over the whole range and one prefix-based over the whole range.
Amended on Sun 28 Jan 2007 08:08 AM by David Haley
USA #6
Wow, Ksilyan, that was really indepth while remaining surprisingly lucid. Well done. In this case, we opted to utilize strcasecmp and went with the solution posted at http://www.fussproject.org/index.php?a=topic&t=1093&p=3997#p3997 but I loved your write up here, I think I may have learned several things from it. :)
USA #7
Thanks. :-) I am about to edit my post and fix a few small things, and clarify some others.

I have a quick question about the post you referred to: what does strcasecmp do? Is it a case-insensitive version of strcmp? Isn't that the same thing as str_cmp?


(str_cmp is a good example of how to not name a function. Who is to guess that the difference between strcmp and str_cmp is that the former is case-sensitive but the second isn't? Grr...)
USA #8
Yes, strcasecmp is the case insensitive version of strcmp. I'm not sure why str_cmp wasn't chosen, perhaps Keberus didn't realize it was case insensitive either. But his fix as posted does work.

Of course, I'm one to tinker, and I decided to try and see if your idea to only do 2 binary searches would work. One for exact, one for prefix, both over the entire range. The results were surprisingly a dismal failure. Here's how I converted skill_lookup:


/*
 * Lookup a skill by name.
 */
int skill_lookup( const char *name )
{
   int sn;

   if( ( sn = bsearch_skill_exact( name, gsn_first_spell, gsn_top_sn - 1 ) ) == -1 )
      if( ( sn = bsearch_skill_prefix( name, gsn_first_spell, gsn_top_sn - 1 ) ) == -1 && gsn_top_sn < top_sn )
      {
         for( sn = gsn_top_sn; sn < top_sn; ++sn )
         {
            if( !skill_table[sn] || !skill_table[sn]->name )
               return -1;
            if( LOWER( name[0] ) == LOWER( skill_table[sn]->name[0] )
             && !str_prefix( name, skill_table[sn]->name ) )
               return sn;
         }
         return -1;
     }
   return sn;
}


I'm puzzled by the result, since this looks valid. But upon bootup, none of the skills from the class or race files would match up. The log filled with spam on every last lookup. It then barfed up on all of the ASSIGN_GSN calls. So it does in fact appear as though doing all of the individual lookups was necessary. But I'd sure love to know why that is.
USA #9
Ah-ha!!

For some reason, str_cmp only returns true/false. It doesn't actually return -1, 0 or 1!

I ran the program through gdb and that's how I discovered this. The binary search was going into the second half of the array, when looking for acetum primus -- it was going forward when it should have been going backwards!



Are you using strcasecmp, or str_cmp?

It would appear that strcasecmp is a C library function that behaves like strcmp, in the sense that it returns -1/0/1.


Now, when I use strcasecmp, I still get some weirdness. It doesn't work.

I will investigate this further tomorrow. Something very fishy is going on; I suspect it has to do with how the array is formed.
USA #10
Heh. See, we always find some of the strangest little mysteries :)

I'm aware that str_cmp only gives a true/false response, but it just hit me as to why that was significant since strcmp and strcasecmp will also give a negative number as a response.

But our fix is using strcasecmp, so I don't see why that's causing a problem with the modified skill_lookup I tried. Obviously it has something to do with how the skill array is constructed but it makes pretty much no sense that you can't just do the binary search on the entire range instead of in 12 little chunks like I have.

It would be nice to get FUSS trimmed down to only use what it needs. Not only is it more efficient, but it's easier to read and maintain when adding new types of skills if you don't have to fiddle with so much code to accomplish it.

Personally I would much rather use a std::map for my own code, but I haven't wrapped my head around how exactly to do that with the skill table and still prevent the stupidity of checking if a value has a valid entry without it creating a blank default. I'm probably over thinking it.
USA #11
Quote:
but I haven't wrapped my head around how exactly to do that with the skill table and still prevent the stupidity of checking if a value has a valid entry without it creating a blank default

What's wrong with the way I showed you using the find method?
Australia Forum Administrator #12
Quote:

I haven't wrapped my head around how exactly to do that with the skill table and still prevent the stupidity of checking if a value has a valid entry without it creating a blank default.


It isn't really stupid.

The header for the map file indicates that the operator[] returns a reference to the item. There is no concept in C++ for a reference to a non-existant object. (You can have a pointer to something that doesn't exist, that is NULL).

Thus, the only way STL can return a reference to a map object using operator[] is for it to create one, if necessary, if it doesn't exist.

In the book "The C++ Standard Library" by Nicolai Josuttis (an excellent book), he comments in the chapter on maps that the operation:



m[key] - Returns a reference to the value of the element with key key.

Inserts an element with key if it does not yet exist.


That is pretty explicit, and as David says, you can use 'find' to see if the element exists, if that is what you want to know.
Amended on Sun 28 Jan 2007 11:59 PM by Nick Gammon
Australia Forum Administrator #13
This small test program shows how you might make a map of skills, and see if one exists:


#include <map>
#include <string>
#include <iostream>

using namespace std;

int main ()
  {
  map<string, string> skills;

  // add a couple of test cases

  skills ["fireball"] = "test1";
  skills ["iceball"] = "test2";

  // iterator for looking up stuff

  map<string, string>::const_iterator it;

  // do lookup 

  it = skills.find ("fireball");

  if (it == skills.end ())
    cout << "not found" << endl;
  else
    {
    cout << "found" << endl;
    cout << "value is " << it->second << endl;
    }

  }


Now if you change the lookup (the find line) to something that isn't there, it echoes "not found".

If it is found, then we can use the iterator "second" value (the value associated with the key) to see what the value is (in this case, "test1").

Maps are very powerful like that, and have a fast lookup. The map doesn't have to be a string-to-string map, the types are specified in the template. For example, you could have a string key (the spell name) but the item being stored could be a structure, or a pointer.
Amended on Mon 29 Jan 2007 12:10 AM by Nick Gammon
Australia Forum Administrator #14
Quote:

(You can also see that for Nick's example of size 100, we needed 7 lookups which is consistent with the numbers above.)


That is gratifying, to see it work. :)

You can check this with an ordinary calculator. Key in:

100 log / 2 log

You will get 6.644.

Round that up to 7, and that agrees with the fact that I found the answer in 7 steps.

Similarly for a table of 1000:

1000 log / 2 log = 9.966

This means that only 10 iterations are needed to find an item out of 1000.

Interestingly, it doesn't matter what base logarithms you use, so using the "ln" key (natural logs) or "log" key (logs to the base 10) give the same answer. As you are dividing one by the other, the base effectively cancels out.
Amended on Mon 29 Jan 2007 12:16 AM by Nick Gammon
Australia Forum Administrator #15
It might be worth pointing out that you can make maps case-insensitive too. Read my post here:

http://www.gammon.com.au/forum/?id=2902

Search for "Case-insensitive maps" on that page. That shows how you can make a map that automatically uses case-insensitive keys, if that is what you want. The same technique applies to all appropriate STL containers (eg. multimaps, sets, etc.).
USA #16
I'll have to look into it all a bit more since changing the skill_table itself in Smaug to a std::map is no small undertaking.

I'll also be taking a look at your case insensitive map since that's probably going to be quite important for keeping sanity :)

I'm assuming that you could do something like this:


skill_type *skill;
skill = skill_map["fireball"];


Where the skill_map is defined as:

std::map<string,skill_type*> skill_map;


Or would I be wrong in that?
USA #17
Yes, but again, that will create the entry if it doesn't exist already. You might want to define a helper function of some kind that does the map.find call, and returns null if not found or the value if found. (Of course, this means you can't distinguish between not having a key, and having it but containing null, but in this case that shouldn't really matter.)
USA #18
Samson, and others interested, I figured out why the single bsearch was not working.



Indeed it had to do with how the skill table was sorted.


If you go to tables.c:639 (or thereabouts), function skill_comp, you will see that it does two if checks based on type:

if( skill1->type < skill2->type )
  return -1;
if( skill1->type > skill2->type )
  return 1;


That made the skill table be sorted by skill type in addition to alphabetical order.

To fix this problem, and have the single bsearch calls work, it suffices to comment out those two lines.

Presto...

At some point, I might time this both ways to see what kind of difference this actually made in terms of time. Still, even if it doesn't make a difference with respect to time, it makes the code a whole lot easier to read.
USA #19
It's always the simplest things that create the biggest problems :)

That's good to know though. But the timing would have to be pretty well worth it to justify ripping out the rest of the code that depends on the gsn sorting being the way it is now. If you look at stuff like find_spell, find_skill, find_tongue, etc. you'll see what I mean.
USA #20
Aww, there's not that much to change, really. And besides I would think that the maintainability increase more than justifies the change. If you'd like, I can submit a patch for SMAUGfuss 1.7 to you in the near future.

First, though, I will do the timing, if anything just out of curiosity.
USA #21
My timing results:

Quote:
New version:
Time elapsed: 14 secs, 233422 usecs

Old version:
Time elapsed: 32 secs, 256077 usecs


Basically it's twice as fast.




Now, some details about my testing.....


My "old version" is the case-insensitive version.

My "new version" is that version, except that it sorts the table only according to name (not according to skill type), and it only does one exact bsearch and one prefix bsearch.

My test:

(1) record start time
(2) Pick a random skill 20,000,000 times and look it up in the skill table, making sure it gets found
(3) record end time

The code, inserted into db.c:


   for( x = 0; x < top_sn; x++ )
      if( !gsn_first_spell && skill_table[x]->type == SKILL_SPELL )
         gsn_first_spell = x;
      else if( !gsn_first_skill && skill_table[x]->type == SKILL_SKILL )
         gsn_first_skill = x;
      else if( !gsn_first_weapon && skill_table[x]->type == SKILL_WEAPON )
         gsn_first_weapon = x;
      else if( !gsn_first_tongue && skill_table[x]->type == SKILL_TONGUE )
         gsn_first_tongue = x;

   // BEGIN TEST CODE

   // do some timing tests
   struct timeval startTime;
   gettimeofday(&startTime, NULL);

   // grab 20,000,000 random skills
   int i;
   for (i = 0; i < 20000000; i++)
   {
       int sn = number_range(1, gsn_top_sn-1);

       const char * name = skill_table[sn]->name;

       assert(skill_lookup(name) != -1);
   }

   struct timeval endTime;
   gettimeofday(&endTime, NULL);

   time_t secs = endTime.tv_sec - startTime.tv_sec;
   suseconds_t microsecs = endTime.tv_usec - startTime.tv_usec;

   printf("Time elapsed: %ld secs, %ld usecs\n", secs, microsecs);

   exit(0);

   // END TEST CODE

   log_string( "Loading classes" );
   load_classes(  );
USA #22
Samson, is there any interest in this change becoming a part of stock SMAUGfuss? Or a patch of any kind?
USA #23
Yes. It would be great to have the skill lookup system simplified.
USA #24
There are definitely a few of us watching this thread to see what will come out of it towards a FUSS fix, Ksilyan. :)
USA #25
I have finished making what appear to be the necessary modifications; I have not fully tested them, though, because I don't have a full SMAUGfuss distribution available (with lots of areas, players, etc.) I see no real reason why it won't work, though. My main concerns have to do with loading and saving players with affects, and herbs. Slot numbers seem to not be an issue. But even those issues are paranoia; I did look through it all and it looks like it should work.

I will be uploading the patch to the SMAUGfuss forums shortly, and hopefully somebody will be able to confirm that it loads seamlessly into an existing world. When I post it, I'll put the link here.
USA #26
Just curious... isn't changing the skill table to NOT be sorted by type going to break the do_practice display? If I remember correctly that depends on the table being in order of type. I don't know anywhere else that relies on this, but that comes to mind.

If that's the only place, and it is indeed a problem, there's probably a decent way around it with this fix.
USA #27
Hmm, that probably will be a problem. I'll have to fix my patch then; that's the kind of stuff I might not notice straight away. :)

But yes, that will be relatively easy to fix.
USA #28
Here is the SMAUGfuss forum post for my patch:
http://www.fussproject.org/index.php?a=topic&t=1109

I fixed the 'practice' issue by creating a parallel skill table, that is sorted by type. It seemed like the easiest thing to do. The parallel table is resorted every time the skill table is changed. I figure that the skill table will change very rarely, and it's more efficient to just cache the sorted-by-type table than to iterate over the skill table several times, once per skill type, or to rebuild a sorted table every time, by storing found skills in temp. lists while iterating over the main table.
USA #29
For an established world, the skill table probably shouldn't be changing often, and then probably only minor changes such as an adjustment here or there or the addition of a new skill or two now and then, but for a world in development, the skill table could have periods of extensive changes happening at a pretty rapid pace. But I suppose if you cache it when it changes, it'd all be good.
USA #30
The caching is just to resort a copy the skill table. So there are two copies at all times: (a) the main one, sorted by name, in which the binary searches are done. (b) the type-sorted one, only used for the practice command at present.

Sorting the skill table does not take much more time than what is needed to run the practice command in the first place. But to maintain the behavior of practice (sorted by type) without having a type-sorted table would be somewhat expensive.

Besides, in order for the new binary search to work, the skill table has to be resorted.


But... hrm. That could be problematic, because if a skill is referenced by number, and the table is resorted, that number could now point to a different thing. That must be why they had that extra segment at the top of the skill list that would be searched linearly in case the skill wasn't found in the rest. I will need to look into this because it could be potentially disastrous. Hrm. Drat.
USA #31
Sorry about the monkey wrench, but better to have all the bases covered. ;)