SVG is a great format for graphics. Unlike the pixel-based formats such as jpeg and png, which store informations about each pixel, SVG relies on geometrical primitives such as curves, colors and coordinates. This results in a much better experiences when images are scaled, making the format ideal for icons and logos.
However, SVG images can also be a major security threat. They are based on XML for storing coordinates but also allow for embedded Javascript code. On a presentation by Mario Heiderich, he managed to make a phone call from a SVG image.
Because of the security threat, SVG images are disabled by default in WordPress. If you do decide to allow them, you need to install a plugin that disables this restriction. Two of the more popular plugins are SVG Support and Safe SVG. After installing one of these plugins, it is recommended you only allow administrators to upload SVGs.