Category Archives: software testing

Testing With Cats?

Our new family member, Sally:

Image

We recently decided it was time to start looking at adding a canine member to our family.  For some background, we have 3 young boys (8 Years old, 2 years old and 9 months) and an adult cat (approximately 13 years old).

Our oldest had a bad experience very young with a playful puppy so is a little hesitant when it comes to dogs. Our 2 year old is obsessed with dogs and goes on walks looking for them all the time. HE also has about 10 stuffed dogs in his possession.

Some of the criteria we used when searching for the dog is as follow:

  • Good with kids: We need a dog that is relatively patient and easy going as with the 3 young kids am sure they will test its patience on a regular basis.
  • Good with cats:  We wanted to make sure the dog we adopted would be good/ok with cats. We didn’t want our cat to be limited to where he could/couldn’t go in house in fear of our new dog.
  • No puppies: Wanted a slightly older dog that while still energetic, a little more mature and calm when needed.
  • Medium sized dog: we didn’t want too large of a dog for space and concerns with knocking kids over, etc.
  • Non-jumper:  This came from our 8 year old. He didn’t want a dog that jumps up a lot as scares him from his prior experience as a smaller child.

We located a dog we were interested in located down in Linton, IN, around a 2 hour drive. She was described as calm and laid back.

Now to the Topic at Hand…

We are now getting to the subject of this blog post. One of the key things before deciding to make the trip down was for them to test her and how she responds to cats. A couple days later, I received a response that they had tested her with cats and that she  tested well and was actually intimidated by the cat they put her in with. So in effect was determined “test passed”. Although she did use some safety language – “I think she will do just fine with cats.”

When we got home and settled in, Dog sees Cat. Cat Sees Dog. Cat takes off. Dog gives chase. Not in an “I’m going to hurt you” way but a “let’s play” way. But, not exactly what we were expecting when heard “tested well with cats”.

As I am in the middle of BBST Test Design training, the functions and variables jumped out to me and motivated me to put together a blog post.

  • Function: Good with Cats
    • A couple Variables stick out: (I am sure folk can find many more in this example).
      • Cat: Is the cat afraid of the dog? Yes or No gives much different results
        • Test Cat: They tested her with a cat that had no fear of dogs and went right up to her. In this case, she cowered and was intimidated.
        • Our Cat: Our cat is skittish of dogs and showed fear by running away. This triggered our dog to give chase rather cower in intimidation.
        • Historical Example: My mom had a dog and 2 cats at 1 time. The 1 cat could care less that the dog was there and would go aboThe other was always scared of the dog. So when we would let them out, he would dart for the fence.  The dog would chase after the scared cat EVERYTIME and never once gave the other much of a bother.
    • Setting:  In shelter vs. at home. At home pre-adjustment vs. At home  and increased comfort level of dog
      • Shelter: She had been there 3 weeks and in a loud environment and adjusting to contact.
      • Our House:  At this point, think the cat variable biggest impact, but will be interesting to see how this impacts her behavior here vs. other areas as she moves from adjusting to our house to feeling like this is her house too.
    • Take Away:
      • This shows the importance of using many techniques to test the different variables in a function/situation.  Superficial coverage to determine “passing” could only give you half the story and/or miss something critical.
      • We may jump to a conclusion on finding a certain pattern and stop short of finding the real issue/answer.
      • Changing any variable can impact the results of test/behavior and then combining multiple variables exposes other potential differences and/or more confidence in consistent behavior.
      • We need to research and figure out the best way to introduce Sally, the Dog, to Will, the Cat, in a way where they can live and interact in peace.  I personally think it would take the cat standing his ground 1 time to set the dog straight.
      • Just because they test out find with one cat does not translate to being great with other cats.
      • Overall she has acclimated to our family very quickly and loves our boys already. Our 8 year old who generally shies away from dogs is on top of the moon and super excited. It is fun to see. She loves to snuggle and walks well on the leash and sleeps in crate without issue.
      • Another area to train her is how she responds/ interacts with our 2 year old. She has made a few warning noises/ moves to our 2 year old if he startles her or is playing around her tail.
Advertisements

Progressive vs. Non-Progressive Bugs

In the past year or so, I have used this comparison when talking with our product and development teams. I highlight the differences in what I term Progressive and Non-Progressive Bugs.  While they are both in the bug category, one is more helpful in gaining insight into the state of the system as well as moving us along the path to even more insight.

Others may have varying terms/definitions for the same thing, but I have found this to be helpful in getting my point across when discussing perhaps a lack of Unit and Integration testing before a build, a late arriving change in the release process, etc.

Non-Progressive Bug:

I refer to a Non-Progressive Bug as those that most likely should have been caught in Unit testing and block additional testing. Yes they are a bug, but they aren’t a bug in the type that provides additional information to the tester on the actual functionality being delivered.

An example would be:

  • Development is working on a Create User module.
  • They build & deploy the code for you to test.
  • When you receive the build and open the Create User module, the Create button is not functional – either disabled or generates an error.

This grinds additional testing to a halt and this bug does not need any special type of skill in testing to come across.  It is a bug that must be logged and fixed to move forward, but doesn’t move the process forward to getting this module ready for release and provide the stakeholder with additional information as to its readiness.

Progressive Bug:

A progressive bug is one where you are able to dig your teeth into the code and really explore the boundaries and the bug detected provides you important information to the readiness, stability, etc. of the code under test. It also provides you perhaps a new pathway to go in your round of testing that you may have overlooked otherwise. It continues to allow you to provide stakeholders relevant data to make the informed decision on state of the feature, the risk associated and additional steps needed to get it to a state where you all feel comfortable releasing it to the public.

An example would be:

  • Development is working on a Create User module.
  • They build & deploy the code for you to test.
  • You begin testing the new Create User module and in testing decide to test it for XSS (Cross Site Scripting) vulnerabilities.
  • Your test uncovers that the module does in fact have vulnerabilities in the code.

This is critical information to have and especially in this day and age of so many security exploits occurring across the globe. This is data you can take to the stakeholders and they then have critical information to make the decision on what to do next. This could spark you to try additional security vulnerabilities and/or allow you to move on to other areas of the module to do more testing to gain more insight/information into the feature under test.

Conclusion:

In conclusion, both of these types of bugs need to be resolved. The difference is Non-Progressive bugs need to be fixed in order for the tester to do the actual testing and begin finding Progressive Bugs that will allow them to provide relevant and accurate information to the stakeholders which in turn allows them to make better formed decisions on direction, readiness for release, etc.


Code Lochness & Code Frost

These terms I came up with as a joke over the last couple of years in regards to adherence to code lock down and code freeze.  We have all been there and will be there again where something (or many) slip into the release post-lock down.

Back in the day I used to lament and complain and use this as a crutch to state I could not be job successfully.  Now I have to come to see it differently and plan for it and adjust our approaches to accommodate late changes as well as educating others what testing can/cannot do for them. I will to get to more on that later, but first the origins of the names.

Code Lochness:

This I feel is a more appropriate name for what we think of “Code Lock down” in our daily lives. This idea came out during many conversations/venting sessions with a developer friend of mine.

The term Lochness is in reference to the mythical Loch Ness Monster.  Like Code Lock Down, the Loch Ness Monster is claimed to exist by many yet has never been seen. They are both mythical creatures.

Code Frost:

This is a more recent addition to my vernacular. The term “Code Freeze” is misleading and is more of a Frost. Code is not frozen, but generally (hopefully) the stream of new code/features coming in to a release under test slows down to a trickle after Code “Frost” hits.

Lessons Learned in Software Testing – Lesson #162 “There are always late changes”

I recently re-read this lesson and jumped out now as have been thinking about this a lot.

This lesson talks about how often times new software is being built and there hasn’t been something like this built and/or used before. There are many reasons listed that drive late changes.

  • All requirements not known when project kicked off and planning was done.
  • Requirements change through design sessions, UAT, etc. and as the team gains more info/insight into the target and intended use.
  • Differing needs of the various stakeholders.
  • Sometimes the work involved takes much longer than expected it will take.

I have seen this time and time again as well as other needs that drive late changes:

  • A customer escalation that is found in production and needs to be fixed right away. The risk of impacting that customer relationship by doing nothing is often times higher than that of coding the fix and deploying to release for testing.
  • A customer commitment with a high $$$ value and/or good publicity can weigh in and be willing to accept a certain level of risk to meet a deadline and satisfy the customer.
  • Similar to first bullet point, the risk of NOT making the fix/enhancement is far greater than that of getting the code in place, tested and out the door to production.

How to Respond:

  • Expect and plan for late changes. Understand why these changes are coming in late and develop a context driven approach on how you will test these changes in the limited time frame and what tools/techniques you can use to aid in your testing. If you have an hour or two, figure out what are the best tests and approaches to explore and provide the most information possible on the state of the fix and its impact on the feature/release.
  • Educate development and leadership on what they can actually expect testing to be able to cover given time & resource availability. Make sure they understand the risks associated with implementing a late change post lock down/near release.
  • Don’t use it as a crutch to NOT do your very best work & effort.

My job is to explore and provide relevant information to all relative stakeholders and continue to educate them on software testing’s role in the overall quality of a product and clarify what folks can/cannot expect from the testing group. We can assure quality, we cannot guarantee issue releases.  It does not mean we stop trying for that goal and in that hopefully our understanding of the issue is accurate, our strategy to attack this issue is sound and that we give nothing less than 100% to the task at hand.

The long story short is we will most likely see the The Loch Ness Monster before we ever see a true, full code lock down. And you know what? I am okay with that (now).


My Internal Keynote Address

Any conference you attend there is generally a keynote (or two) from an industry expert and/or celebrity that ties into the general theme of said conference. Generally, the attendees leave the keynote both educated as well as enthused to learn more on the topic.

This enthusiasm can be short-lived and dropped by the time you have arrived home from the conference or it can have a long-lasting impact in which the attendee is awakened and continues forward post conference to learn and try to implement the lessons they learned.

This got me to thinking, what if I gave myself my own internal keynote address each day/week. The goal would be to find a way to frame the upcoming day/week to both energize myself to conquer the task at hand as well as to continue my pursuits (both professionally and personally). Whether it be an upcoming meeting with a co-worker, a current project I am responsible for we all have those days that are taxing and hard to generate the same level of excitement.

A lot of how your day goes is heavily dependent on your attitude. Do you want that day to be a good day or a bad day? Are you going to see challenges as a road block or as an opportunity to learn something new? These are just a few examples of conversations or keynote addresses on could give themselves.

I will blog more on various testing ideas, concepts and/or strategies, but wanted to start (or resume) on this topic as well as make the following plan over the upcoming calendar year and thus hope then helps others hold me accountable (and for me to be accountable to myself as well).

 

  1. Blog on a routine basis. My initial goal is 1 per week and have set a weekly reminder to myself asking “Have you blogged yet this week?” Hopefully more than 1/week and realize that not every blog post has to be a novel. They can be short and to the point as well.
  2. Engage in testing related conversations on social media (Twitter, LinkedIn, etc.) on a weekly basis.
  3. Continue to read and experiment with new techniques/procedures and work to train my team on these.
  4. Actively participate in the IWST’s (Indianapolis Workshops for Software Testing) and present information as well.
  5. At CAST 2014, develop and sign up to do a Lightning Talk. Also to introduce myself to 10 new people that I don’t know and engage in discussion.

After previous conferences I have come back rolling like a ball on fire, but eventually this fades a bit and fall back to the wayside as real life kicks in and let myself get defeated (or delayed). My goal is to continue on and try to make each week as enlightening and motivating as a week attending a CAST conference does for me. I also strive to help those that work on my team gain the same excitement for their craft as well.

Thanks for reading. I will be working on my takeaways from CAST 2013 in Madison, Wisconsin over the next week.


Five Minutes

I recently attended #CAST2011 out in Lynnwood, WA and on the last day attended James Bach’s Context Driven Leadership tutorial.  I was lucky enough to sit at a table with some very talented and smart individuals – Markus Gartner, Phil McNeely and Elena Houser. Coming back from one of the breaks, Markus asked James what he meant by 5 minutes and in what context (as was stated would start in 5 more minutes).   Given the theme of the tutorial and the fact I seem to find many examples when interacting with my son Jack, this sparked a blog idea that you may or may not find interesting and/or relevant.  

We can all agree that 1 minute = 60 seconds correct? and given that, 5 minutes is 300 seconds?  At face value yes, but using 5 minutes with a toddler is completely different. What 5 minutes means depends entirely on the situation and the context it is used. My son’s reaction to the 5 minutes also ranges depending on the situation – is it something he is eagerly waiting for or is it something he wishes to prolong?  This is getting harder as he continually asks questions and now understands that 1 minute = 60 seconds.

In our household, we use a 5 minute countdown for just about everything as this tends to make whatever transition we are about to make much easier for Jack to adapt and react to.  The first thing folks that watch this, will realize is that our “5 minutes” is rarely ever the same amount of time and generally goes well beyond the 300 second definition above.

Scenario #1: Bedtime

When preparing for bed, we do the 5 minute countdown and one of the biggest factors into what this 5 minutes means is “how late of a start did us (the parents) get in starting the countdown. 

  • If we see that Jack is tired or know has had a long day, we may start the countdown earlier and thus the intervals between the minutes is extended and the 5 minutes can take anywhere from 20 to 45 minutes.
  • If we get off schedule, get home late, etc. and get a late start on countdown, the intervals between minutes reduce greatly and generally are through in a max 10-15 minutes (real time).
  • In each case, despite Jack now knowing 60 seconds is 1 minute, he does not keep us honest by counting to 60 for accuracy as he generally wants to prolong nighttime as much as possible so leaves it to us to call it.

Scenario #2: Travelling

It amazes me that kids somehow inherently know how to ask “Are we there yet?” “How much longer?”  When Jack was younger, could say 5-10 minutes or an even more generic answer “in a little while” and that would suffice.  As stated before, he now understands the concept of 1 minute = 60 seconds which has made it a more risky opposition to an accurate time.  the number of times he asks “Are we there yet?” increase/decrease by his level of excitement getting to the destination.

In one case, I stated 5 minutes and he then proceeded to count to 60  5 times. In this one case I happened to get home right as we he was getting to 60 the fifth time. Of course his 60 is debatable to as he counts in different speeds as well.  The only real consequence is when the trip takes longer than stated and then I have to hear about it the rest of the way until we get to where we are going.

But, this is a case where his response to the time is much different. If he is excited or interested in getting somewhere, he is more cognizant of when it will truly be and doing all he can to speed up the process.  So not only do my wife and I behave differently executing the 5 minute countdown, but Jack also responds differently and his level of awareness of the time changes based on the context of the situation.

In conclusion…

This sort of ties into the risk of gaining a shallow agreement with another. Shallow agreement (paraphrasing) as discussed in the tutorial is “the appearance of agreement when you don’t” and generally happens when we seek common ground too quickly. This can easily lead to misunderstanding each other. 

This is especially important when dealing with someone that is very literal (like a 5 year old) who doesn’t get a “figure of speech.” For example, I have a tendency to respond “In a couple of minutes…” and then he responds later “you said it would only be 2 minutes.” My first response is “No I didn’t” but then realized he was using his defnition of the word couple and that couple equals 2 where I am using it as a figure of speech.  It sets the wrong expectation and thus can lead to confusion and/or issue. How many issues are released to production due to individuals thinking they are on the same page, where they really weren’t and overlook a critical piece of the puzzle?

I appreciate and welcome and any and all feedback as I continue my journey and learning of the context driven school of software testing.


Lessons Learned from a 5 year old

This is in honor of my son, Jack, who turned 5 today. If you are in the testing field and you are interested in the world of patterns and perspectives, there is nothing quite like trying to detect those of a curious and creative toddler.

Here is a pic of Jack at his first White Sox game last year:

Perspectives: Everyone has a different perspective in life and that is definitely true for my son Jack. I wish I had time for a ton of examples, but am amazed when we have a similiar experience and/or encounter and I take it a certain way and then days later Jack will bring up that same moment with a completely different take on it. It makes me really think about my perspective.

  • Default: We discussed this in James Bach’s Tutorials at StarWest 2010 and is something that I have been trying to actively work on and remember to not be biased and stuck with my initial thought/reaction to a subject.
  • Historical: Learn from history and past actions and consequences.  When we are in the beer/wine aisle, he will ask if all the beers are mine as I am a beer drinker. When we are buying wine he knows that white wine is for Yia Yia (my mother) and red wine is for Abu (my step father). He knows that he has seen his Nana (my mother in law) drink both so knows either one could be for her. Although I every once in awhile throw off his pattern by having a glass of wine. So be aware of the exceptions to the rule you are working against.
  • User: Thinking from a user’s perspective: I can’t imaging a toy builder or game creator for kids could possibly think of the various ways in which my son “enhances” the game or what he decides to do with the toys he does have. In many ways, it is similiar to the application my team and I are currently responsible for testing. The permutations and ways in which our users can utilize the application is nearly infinite.

Patterns: And when detecting patterns, you learn a lot playing a board game or even a made up game using his matchbox cars and a race track.  They have a tendency to fit and evolve to enhance his chances of winning, but are generally consistent and plays within the set of his own rules that he come up with.

For example, the latest was if your car landed in a certain spot, you would burn in hot lava. But finding the boundaries where the hot lava stopped  and you were safe was very specific in his mind even if it was not to me.

I really just wanted to celebrate my son today, but this topic has been on my mind as i am really digesting and digging in to perspectives and patterns and continuing to drive my own education in the field with the help of others.

Conclusion:

  • Be careful of relying solely on your default perspective and allow yourself to think differently or how someone else might perceive the same problem.
  • Don’t be quick to force a pattern to be there as while it may look like the pattern, you might miss out on the bigger picture.
  • As well, learn from past successes and mistakes, there is always room to improve and learn.  have you seen this problem before? how did you resolve the problem before? what steps can you take to prevent similiar problems in the future?
  • Think of what the users can and may do with the application and not just what the functional spec or any documentation states. There is a great chance they will use your application in ways you never would have imagined.
  • Think like a 5 year old – Don’t constrain yourself by conventional wisdom and how it “should” be.  Obviously that is simple, but don’t let bias and what others tell you how something is, keep you from exploring and finding new ways to do something.

QA and the Long Snapper

Last year  I was watching  a Chicago Bears game. Their long snapper, Pat Mannelly,  made a huge blunder that turned the momentum of the game in their opponents favor. The ball was deep in their own territory and  were in a punting situation.  He made an audible to do a quick fake snap as he thought the other team had 12 men on the field. This would have been a penalty on the defense, but the 12th man got off the field in time and the audible backfired.

The conditions and setting were not right for this chance to be taken. They were deep in their own territory and the penalty would have given them 5 yards and would still be 4th down. The reward did not outweigh the risk in that situation.

After the game , he was surrounded by the media asking him questions about the play and the decision. He stated: (paraphrasing here) “Well I must have not done my job very well if you guys are sitting here talking to me.”  The key point is the long snapper is not a flashy, popular position and they go day in and day out executing their role effectively and quietly with little fanfare.

How many of us in the testing field have run into this same situation when an issue escapes into production and has a noticeable impact on the user?  Generally the first question out of people’s mouths is “How did this not get caught in testing?” Nevermind the 300+ CRs you did find and resolved prior to release to production. You generally do not get the credit for going about and doing your job well, but any slip or escape and it gets magnified.

We know that finding and detecting EVERY bug and touching all possible permutations of the systm is impossible.  In the past, this lack of credit for what we did find and thet magnification of what slipped through was maddening and a very high point of stress.

While I still take it very personally when something we have worked on generates an issue, I also realize that we are not going to find them all. In our testing cycles, we need to weigh the reward vs. risk, being able to question what is the most critical and what we feel is less critical.

This can sometimes lead to a missed defect that we overlooked or underestimated its criticality. These still hurt, but are easier to swallow knowing that the team was diligent, thoughtful and proactive in their efforts. You get up, dust yourself up, and work to resolve the issue and then remember this lesson for next time.

I guess my point or thought is, that if you are now just entering this field, be prepared to NOT get the credit you deserve as well as getting the blame you do NOT deserve from time to time.  Know the situation you are in, educate yourself with the skills to succeed and apply them when it counts in critical situations.