(Adding my original answer so I can update it.)
There are two ways of answering this question. Firstly, for formats that are known to support encryption (e.g. PDF, ePub, ZIP, etc.), you might expect that characterisation tools can report the presence of encryption. This is true, but in practice many of the tools perform poorly because encryption is used in two ways: as well as full encryption (requiring a specific password to unpack a bytestream), encryption can be used to support obfuscation (where the encryption algorithm does not require a password, or only requires a known, shared, password). For example, PDF uses a default empty string password in some cases, and the JHOVE characterisation tool cannot distinguish between this and full encryption.
The second way of answering this question is as follows - are there any ways of spotting encryption in general, i.e. without understanding the format of the bytestream. The answer to this appears to be 'sometimes'. Methods such as the visualisation of bytestream entropy have been used to spot encrypted regions of files, and so this kind of information-theoretical analysis may be useful in some cases. Formally, however, encryption is statistically indistinguishable from compression, and therefore this approach will lead to a lot of false-positives in the presence of compressed data.
There appear to be very few established tools that perform these tasks.
As @Euan pointed out, there are in fact two approaches to my first way of answering this question. You can use/create analysis software like JHOVE, or you can instead use the 'native' software that implements the format and observe its behaviour (e.g. 'Does Adobe Reader ask for a password?').
UPDATE: During the SCAPE project, the DRMLint tool was developed for this purpose, and is now called Flint.